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ABSTRACT 


This  report  is  the  first  interim  report  of  a  three  year  study  and 
investigation  by  the  University  of  Michigan.  The  primary  objective  of 
this  effort  is  the  study  and  development  of  mathematical  models  of  in¬ 
formation  processing  systems.  The  general  area  of  research  includes 
machine  design,  automata  theory,  and  the  application  of  mathematical 
models  to  problems  in  machine  design.  The  areas  of  research  in  this  re¬ 
port  are  divided  into  these  four  areas  (l)  Automata  Theory  and  Applica¬ 
tions,  (2)  Theory  of  Algorithms,  (3)  System  Analysis,  and  (U)  Combina¬ 
torics  and  Switching  Theory. 


iii 


Table  of  Contents 


Page 

1.  Automata  Theory  and  Applications  .  1 

1.0  Introduction  .  1 

1.1  Permanent  Malfunctions  in  Sequential  Machines  .  3 

1.2  Temporary  Malfunctions  in  Finite  Automata  .  8 

1.3  Formal  Languages .  20 

2.  Theory  of  Algorithms  . .  26 

2.0  Introduction .  26 

2.1  Monotone  Congruence  Algorithms  .  27 

2.2  Discrete  Dynamic  Programming  .  29 

2.3  Branch- and -Bound  Algorithms .  31 

3.  Systems  Analysis  .  35 

3.0  Introduction .  35 

3.1  Storage  Format  Characterization  for  Random  Access  Storage  36 

3.2  Dynamic  Allocation  of  Storage .  1+5 

3-3  Graph  Model  of  Concurrent  Computation  .  47 

3.U  Classification  of  Machines  and  Problems  .  52 

4.  Combinatorics  and  Switching  Theory  .  56 

4.0  Introduction .  56 

4.1  Covering  Problems .  57 

4.2  Threshold  Networks .  60 

4.3  Cellular  Logic .  62 

4.4  Sequential  Circuit  Synthesis .  64 

4.5  Miscellaneous  Problems .  64 

Bibliography 

A.  Publications  by  Laboratory  Personnel  .  66 

B.  Other  References  .  68 


v 


Preface 


This  report  is  a  summary  of  the  results  obtained  from  the  first 
year  of  research  under  contract  A F  30(602) -35^*  In  this  report  many 
mathematical  details  and  proofs  have  been  omitted  in  order  to  obtain  a 
short,  readable  document.  Such  detail  can  be  found  in  the  recent  pub¬ 
lications  of  the  laboratory  personnel.  These  publications  are  listed 
at  the  end  of  this  report  and  should  be  considered  as  part  of  the  re¬ 
search  documentation. 

The  primary  objective  of  this  research  is  the  development  of 
models  of  information  processing  systems  or  models  of  the  functional 
parts  of  information  processing  systems.  The  choice  of  research  areas 
is  influenced  by  existing  problems  in  information  processing  systems 
and  by  the  suitability  of  these  problems  to  abstraction  and  modeling. 
The  general  area  of  research  includes  machine  design,  automata  theory, 
and  the  application  of  mathematical  models  to  problems  in  machine  de¬ 
sign. 

The  report  is  divided  into  four  parts:  (l)  Automata  Theory  and 
Applications,  (2)  Theory  of  Algorithms,  (3)  Systems  Analysis,  and  (k) 
Combinatorics  and  Switching  Theory.  In  some  cases  the  categorization 
of  a  given  topic  in  a  particular  area  is  arbitrary.  In  general^  Auto¬ 
mata  Theory  and  Applications  is  concerned  with  the  behavioral  aspects 
of  finite  state  machines  in  the  abstract  sense  and,  in  particular 
with  relations  between  structure  and  behavior  in  the  case  of  circuit 
malfunctions.  The  relationship  of  abstract  machines  to  programming 
language  is  summarized  and  comments  are  made  with  respect  to  the  need 
for  a  better  understanding  of  semantics.  The  Theory  of  Algorithms  is 
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concerned  with  the  question  of  the  existence  and  the  derivation  of  optimum 

algorithms.  The  abstract  approach  is  taken.  In  Systems  Analysis  models 
are  presented  for  the  organization  of  a  random  access  store  and  for  con¬ 
current  computation.  The  results  of  a  simulation  study  of  the  storage 
allocation  problem  are  presented.  Preliminary  thoughts  on  the  class¬ 
ification  of  machines  and  problems  is  included.  Models  of  the  control 
process  are  being  studied  but  this  research  is  in  the  preliminary  stage 
and  is  not  included  in  this  report.  Combinatorics  and  Switching  Theory 
is  concerned  primarily  with  covering  problems  and  recent  results  in  the 
area  of  threshold  networks. 

A  comprehensive  literature  survey  was  undertaken  early  in  the 
course  of  this  research.  A  detailed  review  of  this  literature  is  not 
included  in  this  report  but  mention  is  made  to  pertinent  references 
and  other  research  efforts  of  interest  to  this  program. 

Professor  Harvey  L.  Garner  has  served  as  the  project  director. 

The  senior  research  staff  has  consisted  of  Professor  Richard  F.  Arnold, 
Professor  Eugene  L.  Lawler,  and  Professor  Richard  M.  Karp.  The  research 
staff  has  consisted  of  P.  Dauber,  J.  DiGiuseppe,  K.  Garrison,  R.  Gonzalez, 
J.  Meyer,  V.  Powers,  G.  Putzolu,  R.  Reiter,  D.  Wood,  and  R.  Zauel. 
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1.  Automata  Theory  and  Applications 
1.0  Introduction 

In  the  decade  or  so  that  automata  theory  has  been  a  recognized 
field  of  study,  great  strides  have  been  made  in  the  discovery  of 
theoretical  results  and  in  the  explication  of  the  interfaces  with  border¬ 
ing  disciplines.  Among  the  notable  achievements  have  been  (l)  a  thorough 
analytic  treatment  of  finite-state  systems  and  (2)  the  development  of 
an  elegant  and  coherent  theory  of  formal  languages  and  of  the  automata 
that  can  be  used  to  generate,  manipulate,  and  recognize  them.  These 
two  bodies  of  theory  seem  to  have  particularly  great  potential  for  appli¬ 
cation  in  the  analysis  and  design  of  computer  systems. 

Our  main  research  effort  in  finite-state  systems  has  been  concerned  with 
error  properties  (aside  from  the  question  of  synthesis,  discussed  in 
section  4),  and  two  complementary  points  of  view  have  been  adopted. 

Under  one  point  of  view  a  malfunction  is  considered  to  be  temporary, 
leaving  the  state  transition  structure  of  the  machine  unaltered.  A 
natural  question  to  ask  is:  "What  is  the  probability  that  the  system 
will  be  automatically  restored  to  its  proper  state?"  Under  the  other 
point  of  view,  a  malfunction  is  considered  to  be  permanent,  changing 
the  state  transition  structure  in  some  significant  way.  Natural  ques¬ 
tions  to  ask  are:  "How  is  the  new  system  related  to  the  original?  Can 
the  inputs  be  modified  or  coded  sc  that  the  desired  behavior  is  realized 
in  spite  of  the  malfunction?  Can  the  system  be  designed  in  such  a  way 
that  malfunctions  in  the  system  will  produce  only  tolerable  changes?" 

A  study  of  the  temporary  type  of  malfunctions  conducted  by  P.  Dauber 
is  reported  in  Section  2.  The  initial  formulation  of  a  study  of 
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permanent  malfunctions  being  conducted  by  J .  Meyer  are  presented  in 
Section  1. 

The  theory  of  formal  languages  has  contributed  to  a  better  under¬ 
standing  of  the  necessary  properties  of  programming  languages  and  to 
more  systematic  methods  of  specification  and  of  processing.  Various 
unsolved  problems,  including  an  inadequate  treatment  of  semantics, 
appear  to  limit  further  applications  in  the  design  and  programming  of 
computers.  These  problems  are  reviewed,  and  an  appraisal  of  the  field 


is  made  in  Section  3- 
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1.1  Permanent  Malfunctions  in  Sequential  Machines 

The  particular  point  of  view  adopted  in  treating  the  subject  of  mal¬ 
functions,  failures  or  errors  depends  strongly  on  the  class  of  errors  being 
considered,  where  different  error  classes  reflect  different  interpretations 
as  to  how  and  where  the  errors  are  caused.  If  the  errors  are  regarded  as 
arising  externally,  that  is,  at  the  input  to  the  machine,  then  the  inter¬ 
nal  structure  is  not  affected  by  error.  Thus,  the  effects  of  input  error 
on  the  behavior  of  a  particular  sequential  machine  can  be  classified  and 
analyzed  in  terms  of  its  structure  (transition  function),  a  structure 
which  one  can  assume  to  be  fixed  under  all  possible  errors.  Represen¬ 
tative  of  this  type  of  error  analysis  are  the  contributions  of  Neumann 

-r 

[b31],  Winograa  [b36],  and  most  recently  Harrison  [bl8] . 

Another  general  class  cf  errors  are  those  which  are  regarded  as 
being  the  result  of  Internal  malfunctions.  In  the  case  where  the  cause 
of  the  error  is  transient  or  temporary  in  nature  and  results  in  the 
machine  assuming  an  erroneous  state,  the  analysis  of  such  errors  is  close¬ 
ly  related  to  that  of  the  input-error  interpretation.  This  class  of 
errors  can  again  be  related  to  a  fixed,  deterministic  structure  since, 
following  the  disturbance,  the  operation  is  assumed  to  be  error  free. 

Thus,  this  type  of  error  can  equally  well  be  regarded  as  being  caused 
externally  by  some  input  error.  The  questions  asked  differ,  however, 
since  they  relate  directly  to  state  errors  rather  than  to  the  nature  cf 
the  inputs  that  might  have  caused  them.  This  is  the  point  of  view  adopted 
by  Hartmanis  and  Stearns  [b20]  and  Dauber  [  a4  ]  in  their  investigations 
cf  this  type  of  state  transition  error. 
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Tae  nature  of  intemallj  caused  errors  has  also  been  studied  extensively 
from  a  synthetic  point  cf  view  j that  is,  in  terms  o*  'imitive  elements  whose 
interconnection  realizes  some  sequential  network.  Here,  the  primitive  ele¬ 
ments  (which  ’.'.jve  been  regarded  as  logic  primitives  and  neurons  at  one 
extreme  and  large  subsystems  at  the  other)  are  assumed  to  have  a  certain 
probabilistic  behavior.  The  analysis  is  then  concerned  with  questions 
that  relate  the  probabilistic  nature  of  the  primitive  elements  to  that  of 
interconnected  networks  of  these  elements.  This  was  the  point  of  view  taken 
in  the  classic  contributions  of  von  Neumann  [b35]  and  Shannon  and  Moore  [b28]. 
Since  their  papers  appeared,  much  of  the  effort  devoted  to  the  study  of 
reliability  and  redundancy  in  switching  networks  and  computing  systems  has 
reflected  this  particular  method  of  analysis  [ b23] . 

The  reason  for  introducing  this  short  summary  regarding  various 
classes  of  errors  and  methods  with  which  they  have  been  investigated  (the 
summary  is  in  no  sense  complete)  i6  to  provide  a  means  of  comparing  the 
class  of  errors  we  propose  to  investigate  here  with  certain  of  those  al¬ 
ready  studied.  The  type  of  errors  we  wish  to  consider  here  can  be  re¬ 
garded  as  being  due  to  internal  failures  which  are  permanent  in  nature. 

Unlike  the  class  of  externally  caused  errors  or  the  class  of  internally 
caused  transient  errors,  we  can  no  longer  assume  that  the  structure 
of  the  machine  is  invariant.  On  the  other  hand,  we  would  like  to  avoid 

restricting  ourselves  tc  a  particular  class  of  switching  elements  as  is 

/ 

done  in  the  synthetic  approach.  This  is  not  to  say  that  a  more  general 
analysis  would  have  no  bearing  on  synthesis  problems.  We  feel  that  by 
relating  the  effects  of  permanent  failure  directly  to  structure  one  can 
maintain  a  certain  degree  of  generality  and  yet  provide  synthesis  procedures 
that  are  applicable  to  various  classes  of  switching  networks.  We  will 
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have  more  to  say  in  this  regard  once  we  have  outlined  the  basic  framework 
and  objectives  of  our  proposed  research. 

We  are  given  a  finite-state  sequential  machine 

M  =  (2,  A,  Q,  X,  5) 

with  inputs  2,  outputs  A,  states  Q) transition  function  S,  and  output 
function  X  defined  in  the  usual  way.  We  now  suppose  that  in  some  physical 
system  represented  by  M  there  is  a  permanent  malfunction  which  permanently 
alters  the  system  but  results  in  a  configuration  that  still  behaves  as  a 
sequential  machine.  We  can  then  represent  the  result  of  the  failure  as 
a  second  machine 


M'  =  (2,  A,  Q',  V,  6’) 

where  the  states  Q',  the  transition  function  6',  and  the  output  function 
X '  of  the  failed  machine  are  related  in  some  way  to  the  original  machine 
M.  A  more  precise  statement  of  this  relationship  depends,  of  course,  on 
more  detailed  knowledge  as  to  how  the  system  failed. 

In  relating  M  to  H'  we  choose  to  restrict  our  attention  here  to  failures 
that  occur  in  the  memory  portion  of  the  physical  system.  This  restriction 
is  motivated  by  the  fact  that  it  is  memory  which  distinguishes  nontrivial 
sequential  machines  from  purely  combinational  systems.  The  latter  have 
been  investigated  rather  thoroughly  with  regard  to  error  susceptibility 
but  few  of  the  results  apply  when  memory  (with  feedback)  is  introduced. 

The  restriction  also  has  the  advantage  that  the  function  of  memory  is 
the  same  from  machine  to  machine,  that  is,  to  store  the  information  pre¬ 
sented  at  the  memory  input. 

In  a  sequential  machine  the  transition  function  represents  both 
decision  and  memory  processes  in  that  we  interpret  £>(q,o)  to  be  the  "next" 


state  given  the  "present"  state  is  q  and  the  "present"  input  is  c. 

To  distinguish  the  functions  of  memory  and  decision  we  can  let 
S  =  6<u  (the  functional  composition  of  6  and  u,  first  applying  §) 
where  S(q-a)  is  the  memory  input  and  represents  a  purely  combinational 
process  and  u  is  the  memory  function  representing  the  storage  of 
S(q.cr) .  In  case  the  memory  operates  properly,  u  is  simply  the  identity 
function  on  the  set  of  states  Q.  Accordingly,  if  memory  inputs  are 
stored  improperly  as  the  result  of  failure,  i±  is  some  function  other  than 
the  identity  function.  To  insure  that  the  result  of  the  failure  is  stable 
in  the  sense  that  the  machine  does  not  oscillate  between  states,  we 
require  in  addition  that  u  be  an  idempotent  function.  We  call  such  func¬ 
tions  failures  of  the  machine  M  and  for  the  failed  machine  M'  we  have 
Q'  -  u(Q),  6'  =  6-pi  and  X'  =  X  restricted  to  u(Q)-  This  then  is  the 
basic  framework  within  which  we  intend  to  study  permanent  failures. 

Within  such  a  framework  a  number  of  interesting  questions  present 
themselves  in  a  rather  natural  way.  In  physical  situations  where  one  is 
unable  to  or  does  not  choose  to  repair  the  system,  one  is  interested  in 
failures  under  which  the  terminal  behavior  is  unaltered  by  the  change  in 
structure.  Thus  we  want  to  determine  the  conditions  on  a  failure  ^ 
under  which  the  failed  machine  is  behavior ally  equivalent  to  the  original. 
Also  of  practical  importance  are  situations  where  internal  repair  is  not 
feasible  but  one  is  able  to  recode  input  and  output  information  in  an 
attempt  to  re-establish  proper  operation.  This  would  occur,  for  example, 
with  airborne  or  spaceborne  computers  that  receive  from  and/or  transmit 
to,  manned  installations.  In  this  case  we  want  to  know  the  conditions  on 
H  under  which  M'  can  simulate  M.  We  may  also  ask  questions  about  preserving 
certain  properties  of  the  behavior  that  correspond  in  some  sense  to  partial 


success  of  the  system. 
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With  regard  to  failures  themselves,  we  would  like  to  determine  those 
relations  on  a  set  of  failures  that  will  connect  the  known  properties  of 
some  given  failure  with  those  related  to  it.  For  example,  if  we  know 
that  some  failure  p  preserves  behavior  in  one  of  the  above-mentioned  ways, 
what  relation  or  relations  on  the  set  of  failures  will  determine  other 
failures  having  the  same  property?  Regarding  the  failures  simply  as  func¬ 
tions,  how  do  well-known  operations  and  relations  correspond  to  the  in¬ 
tended  interpretation?  What  subsets  of  failures  correspond  to  various 
physically  motivated  restrictions  as  to  how  memory  elements  fail?  Can 
these  subsets  be  generated  in  some  way  from  even  smaller  sets?  The  last 
two  questions  are  related  directly  to  the  synthesis  problem,  which  we 
regard  as  an  important  part  of  the  investigation.  In  the  proposed 
framework,  this  problem  takes  the  following  form.  According  to  the 
assumed  nature  of  the  memory  elements  and  how  they  fail  in  a  memory  sys¬ 
tem,  one  can  determine,  for  a  given  size  memory,  the  functions  corres¬ 
ponding  to  possible  physical  failures.  This  can  be  done  without  know¬ 
ledge  as  to  how  the  remainder  of  the  system  is  implemented.  Then  given 
a  reduced  machine  which  satisfies  the  specified  behavioral  requirements, 
the  synthesis  problem  is  translated  to  that  of  a  many-to-one  state  assign¬ 
ment  which  satisfies  the  desired  behavioral  requirement  (e.g.  equivalence 
or  simulation)  on  machines  resulting  from  failures  in  the  set  under  con¬ 
sideration.  As  such  an  assignment  may  not  always  be  possible;  questions 
as  to  the  conditions  under  which  algorithms  exist  are  also  important. 

In  the  process  of  answering  these  questions  we  should  also  obtain  a 
more  thorough  knowledge  as  to  general  relationships  between  structure  and 
behavior,  and  should  be  able  to  relate  failures  to  other  aspects  of  machine 
theory,  especially  decomposition  theory.  The  main  objective,  however,  is 
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to  determine,  as  a  function  of  the  complexity  of  behavior,  the  complex¬ 
ity  of  structure  needed  for  a  specified  degree  of  error  insensitivity. 

In  the  case  of  permanent  failures,  such  questions  have  not  been  answered 
by  the  synthetic  approach,  whereas  we  feel  they  can  be  answered  within 
the  proposed  framework. 

1.2  Temporary  Malfunctions  in  Finite  Automata 

This  problem  arose  from  an  attempt  to  make  a  general  study  of  relia¬ 
bility  in  computer -like  machines.  Machines  of  this  type  may,  due  to  a 
bad  input  tape,  a  temporary  malfunction  of  a  diode,  or  for  some  other 
reason,  enter  an  incorrect  state .  The  machine  may  then  under  the  influ¬ 
ence  of  the  input  tape  yield  incorrect  outputs.  However,  if  the  input 
tape  takes  both  an  incorrect  state,  entered  due  to  the  malfunction,  and 
the  correct  state  to  the  same  next  state,  then  subsequent  outputs  will 
be  correct.  This  will  be  called  correcting  the  error. 

The  classic  results  of  von  Neumann  ]  apply  only  to  networks 
without  feedback.  Thus  a  malfunction  only  causes  the  networx  to  be  in 
the  incorrect  state  for  a  bounded  length  of  time.  With  feedback  a  mal¬ 
function  can  cause  an  error  which  may  persist  forever.  Fortunately,  not 
all  errors  are  of  this  type.  Some  errors  are  of  the  type  that  can  per¬ 
sist  only  for  a  bounded  time.  Some  errors,  although  they  can  persist 
infinitely  long,  have  a  probability  one  of  being  corrected  as  the  tapes 
get  longer.  Thus  "almost  all"  of  the  "long"  tapes  correct  the  error. 

Thi6  phenomenon  has  been  studied  by  Dauber  [  a5  , a6  ].  The  results 
of  this  research  are  summarized  here.  First  the  problem  is  formulated 
in  terms  of  the  theory  of  automata.  In  the  formulation  which  follows  we 
are  concerned  only  with  states  and  outputs  are  ignored.  This  transition 
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system  is  sufficient  for  this  study. 

Definition  1 

A  finite  automaton  M  is  a  triple 
M  =  (Q,  2,  6) 

where  Q  is  a  finite  set  with  elements  q^  (set  of  states); 

2  is  a  finite  set  with  elements  (input  alphabet); 

6  is  a  function  from  Q  X  Z  -*  Q  (next  state  function)  • 

If  we  are  thinking  of  the  finite  automaton  as  a  model  of  a  digital 
device,  we  can  associate  the  states  of  the  digital  device  with  the  state 
set  Q ,  the  input  symbols  of  the  device  with  2.  Then  the  manner  in 
which  the  device  changes  states,  when  it  receives  an  input  is  associated 
with  5. 

Definition  2 

(a)  An  error  in  a  finite  automaton,  M,  is  a  pair  of  states 

(b)  An  error  (q^,q^)  is  corrected  by  a  tape  t  ("tape"  is 

synonymous  with  "input  sequence")  if  and  only  if 
^(q^t)'  =  B(qj,t). 

We  can  think  of  an  error  (q^qj)  as  the  situation,  when  due  to  a 
previous  malfunction,  the  automaton  is  in  state  q^  and  should  be  in  state 
qj  or  vice  versa,  (it  is  obvious  from  the  definition  "corrected"  that 
these  situations  are  equivalent.)  If,  by  the  above  definition,  an  error 
is  corrected,  then  from  that  time  on  the  output  must  be  correct.  However 
if  an  error  is  not  corrected  we  may  still  have  more  incorrect  outputs. 


10 


In  this  work  we  will  consider  a  random  source,  which  generates  se¬ 
quences  and  drives  the  automaton.  A  random  source  S  is  a  set  (P^)  of 
probability  distributions  Pn(x),  the  probability  of  the  n-length  string 
x.  S  has  property  P  if  there  is  a  real  k  >  0  such  that  Pn+1(xa)  >  k  Pn(x), 
for  all  x  and  cr. 

Lefinition  3 

Let  S  be  a  random  source  with  property  P  and  output  symbols  Z, 

and  let  M  =  (Q,  Z,  6)  be  a  finite  automaton  driven  by  S.  For 

an  error 

E  =  (q  ,q.)  we  define  the  following: 
i  1 

g 

(a)  =  probability  of  the  set  of  tapes  of  length  l 

which  correct  the  error  (q^q^). 

(fc)  7S(qi>q1)  =  7|(qi/qJ- 

Now  let  us  consider  the  following  classification  of  errors  in  a 
finite  automaton  M  being  driven  by  a  source  S  as  above. 

Definition  4 

An  error  E  =  (q^q^)  is 

(a)  definite  if  and  only  if  there  is  an  I  such  that 

h®  ■  1. 

(b)  finite  if  and  only  if  y  (E)  =  1. 

(c)  correctable  if  and  only  if  y  (E)  >0. 

(d)  non-correc table  if  and  only  if  y  (E)  =0. 

Intuitively,  these  classifications  have  the  following  meanings.  If 
an  error  is  definite  then  there  is  a  fixed  length,  /,  such  that  all  se¬ 
quences  of  length  l  cr  greater,  correct  the  error.  On  the  other  hand,  if 
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there  is  not  such  a  fixed  length,  out  as  we  consider  longer  and  longer 
sequences,  a  larger  and  larger  percentage  of  sequences  correct  the  er¬ 
ror,  and  if  in  the  limit  one  hundred  percent  cf  the  sequences  correct 
the  error,  then  we  have  a  finite  error. 

We  will  now  give  some  fundamental  properties  of  errors  which  will 
show  the  connection  between  the  concepts  of  correctable  and  finite  errors. 

Theorem  1 

The  set  cf  finite  errors  in  an  automaton  M  driven  by  a  source 

with  property  P  induces  a  partition  on  the  set  cf  states. 

That  is,  there  is  a  partition  rt  on  the  states  of  M  so  that 

E  =  (q^q^)  is  finite  if  and  only  if  qi  s  q^n^,). 

The  next  theorem  will  show  the  strong  connection  between  errors 
which  are  correctable  and  errors  which  are  finite.  We  will  use  the 
cross  product  of  sets  which  has  the  following  meaning: 

AxB  =  [ (a. ,b . )  I  a.€A  and  b  .  eB} . 

i  j  1  i  j 

Theorem  2 

Let  C  C  M  X  M  be  the  relation:  (q.,q.)eC  if  and  only  if 

^  J 

,q.)  is  a  correctable  error.  Then  an  error  E  =  (q.,q. 

0  1  0 

is  finite  if  and  only  if  (q.jq.)€C  and  for  all  tapes  t, 

i  j 

(S(qj_,t),  5(qj,t))€C. 

Since  the  concept  of  an  error  being  correctable  is  not  dependent  upon 
the  source,  the  above  theorem  tells  us  that  as  long  as  we  are  dealing 
with  only  the  class  of  sources  that  have  property  P,  the  property  of  an 
error  being  finite  is  also  independent  of  the  source.  In  the  discussion 
which  follows  the  term  "source"  will  refer  to  a  source  with  property  P 
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unless  we  explicitly  say  otherwise.  Thus  we  will  call  an  error  a  finite 
error  if  it  is  finite  for  some  source  (hence  all  sources)  with  property 
P,  and  we  will  call  itp  the  finite  error  partition.  Likewise  we  will 
drop  the  superscript  on  7  denoting  the  source.  We  will  call  an  error  a 
nontrivial  error  if  it  is  not  of  the  form  (q^,q^). 

Theorem  2  provides  a  conceptual  connection  between  the  relation  C 
and  the  partition  n^,.  The  next  theorem  is  a  stronger  characterization 
of  this  connection. 

We  will  use  the  canonical  ordering  on  partitions.  That  is, 

«1  >  *2  ('Ll  -  ^  5 

Theorem  3 

itp  is  the  coarsest  partition  (i.e.  the  largest  under  the 
canonical  ordering)  wiuh  the  substitution  property  such 
that 

qi  5  qj  y  (q^q^ec. 


State  behavior  realization  as  defined  by  Hartmanis  and  Stearns  fcl9,b22] 
is  a  very  strong  type  of  realization.  One  meaning  is  that  if  one  digital 
circuit  is  represented  by  an  automaton  and  another  by  an  automaton  Mg 
and  if  state  behavior  realizes  Mg  then  no  matter  what  coding  of  the 
outputs  we  put  on  the  second  system  there  is  a  coding  of  outputs  of  the 
first  so  that  they  both  do  the  same  thing. 


s 
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Definition  5 

Let  =  (Q1,  2,  5),  Mg  =  (Qg,  5’)  and  X  :  q±  X  2->S'. 

Then  the  series  connection  of  with  Mg  with  connecting 
function  X  is  the  automaton  M  =  (Q^  X  Qg,  2,  8")  where 
8"  is  defined  as  follows: 

=  (stays)*  &’(qy  ^q^q)))* 

We  will  say  that  a  finite  automaton  M  can  be  state  behavior  realized 
by  a  series  connection  of  finite  automata  and  Mg  if  there  is  a  connec¬ 
ting  function  X  such  that  M  is  state  behavior  realized  by  the  series  connec¬ 
tion  of  and  Mg  with  connecting  function  X. 

Again,  in  terms  of  digital  systems,  this  means  that  if  D  ,  Dg,  and 
D^  are  three  digital  systems  represented  by  finite  automata  M^,  Mg,  and 
M^  and  is  state  behavior  realized  by  a  series  connection  of  Mg  and 
My  then  instead  of  building  D^,  we  can  build  Dg  and  D^  and  connect 
them  as  follows: 


In  this  case  X  corresponds  to  the  coding  of  the  outputs  of  Dg. 

The  following  theorem  is  a  consequence  of  Theorem  3  and  a  well-known 
result  of  Hartmanis . 

Theorem  ^ 

If  M  is  a  finite  automaton  with  a  finite  partition  then 
M  can  be  state  behavior  realized  by  a  series  connection  of 
two  automata  M^  and  Mg,  where  all  errors  in  Mg  are  finite, 
and  M^  has  no  nontrivial  finite  errors. 


The  following  example  demonstrates  these  theorems. 


Let  M  =  ((a,b,c,d,e),  (0,l),  6)  where  6  is  the  mapping  shown  below. 


6 

0  1 

a 

b  d 

b 

a  d 

c 

a  _b 

d 

b  d 

e 

a  d 

It  is  easy  to  show  that 

C  =  l(a;d),(d,a),(b,c),(c,b),(e,a),(a,e),(e,d), (d,e), (b,e), (e,b), 

(c,e),(e,c),  (a-,a),(b,b),(c,c),(d,d),  (e,e))  . 

There  are  four  equivalence  relations  with  the  substitution  property 
contained  in  C . 

'=  (a,b,c,d,e) 

*2  =  (ajd,  b,  c,  e} 

=  [a,  b,’c ,  d,  e) 

=  (aTd,  b7c;  e) . 

The  coarsest  one  is  it^.  Thus  the  only  nontrivial  finite  errors  are 

((0jd),  (d,a),  (b,c),.(c,b)) . 

The  following  theorem  is  another  corollary  of  Theorem  3.  It  was 
first  proved  in  another  context  by  f ' lbert  and  Moore  bl4  ] . 

Theorem  5 

All  errors  in  an  automaton  M  are  finite  if  and  only  if  M  has 
a  reset  tape.  (A  tape  t^s  a  reset  tape  if  &(qi,t)  is  inde¬ 
pendent  of  q,.). 


Let  us  now  look  at  another  example  to  show  the  use  of  this  theorem. 
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Example  2 

Let  M  =  ((a,b,c,dl,  (0,1),  5)  where  8  is  shown  below. 


1_ 

c 

d 

b 

b 


It  Is  easy  to  see  that  all  the  errors  are  correctable .  Hence  by  Theorem 
3,..  .rtj,  =  (a,b,c,d)  and  all  errors  are  finite.  Upon  examination  it  can  be 
seen  that  the  tape  000  is  a  resex  tape  since  8(qi,000)  =  d  regardless  of 
q^.  Thus  we  could  have  found  that  all  errors  are  finite  by  applying 
Theorem  5. 

In  the  course  of  studying  errors  in  finite  automata,  it  became  ob¬ 
vious  that  many  of  the  error  properties  (as  well  as  other  properties)  of 
finite  automata  were  more  easily  discussed  in  terms  of  the  semigroups 
of  the  automata  [b25  ,  b26j .  We  summarize  some  of  these  results  here, 
without  repeating  the  exposition  of  abstract  semigroups  contained  in  the 
complete  report. 


Definition  6 

Let  M  =  (Q,  2,  8)  be  a  finite  automaton.  S^,  the  semigroup 
of  the  finite  automaton  M,  is  the  semigroup  whose  elements 
are  transformations,  mapping  the  set  of  states  Q  into  itself, 
induced  by  the  next  state  function  8.  The  multiplication 
operation  s^.s^  is  the  composition  of  s^  and  s^. 


It  is  clear  that  the  multiplication  operation  in  the  above 
definition  is  associative  and  that  the  set  of  its  elements  is  closed 
under  multiplication.  Hence  is  indeed  a  semigroup,  as  desired. 
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Theorem  6 

Let  E  be  an  error  in  a  finite  automaton  M.  Then  E  is 

1.  correctable  if  and  only  if  E  is  corrected  by  some 
minimum  idempotent  of  S^. 

2.  finite  if  and  only  if  E  is  corrected  by  every 
minimum  idempotent  of  S^. 

We  will  now  state  an  immediate  corollary  to  this  theorem.  We 
will  use  ng  to  indicate  the  partition  induced  by  the  mapping  associated 
with  the  semigroup  element  s.  That  is,  two  elements  are  in  the  same 
class  of  the  partition  if  and  only  if  the  mapping  s  takes  them  into 
the  same  element. 

Theorem  7 

An  error  E  =  (q  ,q.)  in  a  finite  automaton  M  is 

1.  correctable  if  and  only  if  q^  =  q^  (jt^  )  for  s^,  some 
minimum  idempotent  of  S^. 

2.  finite  if  and  only  if  q.  =  q.  (  fl  «  )  where  {s. }  is 

1  J  {s  }  £i  1 

the  set- of  minimum  idempotents  of  Sw. 

M 

It  follows  that  the  partitions  associated  with  the  minimum  idem¬ 
potents  of  completely  characterize  the  error  properties  of  the 
automaton  M.  If  the  partitions  associated  with  the  minimum  idempotents 
are  known,  then  we  know  which  errors  are  correctable,  which  are  noncor- 
rectable,  and  which  are  finite. 


Definition  7  (Perles,  Rabin,  and  Shamir,  b32]) 

A  finite  automaton  M  =  (Q,2,6)  has  a  k-definite  move  function 
if  and  only  if  for  all  sequences  ...  of  k  letters  from 
2;  5(qt,  ...  Ck)  6(q^,  a±  . . .  afc)  for  all  q^q^  in  M. 

Note  that  this  implies  that  5(q^,  ...  cr^)  =  6(q^,  ak+1,  •  ••, 

...  a^) .  We  will  informally  call  a  finite  automaton 
k-definite  if  its  move  function  is  k-definite. 

The  tie  up  of  definite  errors  and  definite  automata  is  clearer  after 
the  following  two  theorems  which  are  due  to  Hartmanis  and  Stearns  t>20  ] . 

Theorem 8 

A  finite  automaton  M  =  (Q,  2,  6)  is  definite  if  and  only  if 
all  its  errors  are  definite. 


Theorem 


If  M  is  a  finite  automaton,  then  M  can  be  decomposed  into  a 


series  connection  of  two  finite  automata  and  Mg  as  shown 
below  with  all  errors  in  Mg  being  definite  and  no  nontrivial 


error  in  being  definite.  Hence  Mg  is  a  definite  automaton. 


This  theorem  follows  from  the  fact  that  there  is  a  partition  with 
substitution  property  on  the  states  of  M  with  the  property  that  an  er¬ 
ror  E  is  definite  if  and  only  if  The  next  two  definitions  and 

theorem  give  a  characterization  of  the  semigroups  associated  with  defi¬ 


nite  automata. 
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Definition  8 

Let  S  =  (S,*)  be  a  semigroup.  Then,  an  element  z  of  S  is 
a  right  zero  if  and  only  if  for  all  seS,  s-z  =  z. 

Definition  9 

We  will  say  that  M  =  (Q,Z,B)  is  a  union  of  the  finite  auto¬ 
mata  =  (Q^,  £,  6^  1=1,..., k  if  the  following  conditions 
hold: 

(1)  i±j  D  Qj  is  empty. 

(2)  U  Q .  =  Q  . 

i 

(3)  =  B  restricted  toQ^. 

Theorem  10 

Let  M  =  (Q,  Z,  &)  be  a  finite  automaton.  Then  the  following 
two  conditions  are  equivalent: 

(1)  M  is  a  union  of  definite  automata. 

(2)  contains  a  universal  minimum  right  ideal  U  such  that 

all  elements  of  U  are  idempotent  and  all  the  idempotents 

of  S„.  are  in  U. 

M 

Since  a  finite  automaton  is  a  union  of  definite  automata  if  and 
only  if  all  its  errors  are  definite  or  non -correct able  we  know  a  method 
of  checking  for  this  condition  from  the  semigroup.  The  next  two  theorems 
which  are  corollaries  of  this  theorem  give  us  alternate  checks. 

Theorem  11 

M  is  a  union  of  definite  automata  if  and  only  if  every  idem- 
potent  of  is  a  right  zero. 
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Theorem  12 

M  is  a  union  of  definite  automata  if  and  only  if  the  set 
of  idempotents  of  is  a  minimum  right  ideal. 

Theorems  10,  11,  and  12  can  be  applied  to  give  more  results  on 
definite  errors.  For  er  .tuple,  a  finite  automaton  is  such  that  all  its 
finite  errors  are  definite  errors  if  and  only  if  there  are  no  idempotent 
elements  outside  its  kernel.  Also,  it  can  be  shown  that  linear  automata 
have  only  errors  which  are  either  non-correctable  or  definite.  The 
above  results  tell  us  something  about  the  structure  of  semigroups  of 
linear  automata. 

The  complete  paper  by  Dauber  contains  a  number  of  extensions  of 
the  theory,  which  will  not  be  discussed  here.  However,  before  conclud¬ 
ing,  a  few  words  should  be  said  about  the  applicability  of  the  model. 

There  ere  three  basic  points  which  must  be  examined:  the  use  of 
a  finite  automaton  as  an  error  model  for  a  digital  system,  the  defini¬ 
tion  of  correctability,  and  the  use  of  a  random  source  driving  the 
finite  automaton.  If  we  are  considering  synchronous  digital  systems 
with  fixed  memory  capacity,  then  finite  automata  are  good  models.  With 
memory  which  is  extendable  but  bounded,  they  are  still  good  models. 
However,  if  we  consider  the  memory  to  be  arbitrarily  extendable  then  we 
would  need  a  model  with  an  infinite  number  of  states.  As  was  shown  in 
Dauber  [a6],  the  results  given  here  do  not  carry  over  to  this  case. 

The  definition  of  a  tape  correcting  an  error  also  is  not  a  bad 
model  if  we  are  careful  to  keep  in  mind  what  it  means.  It  does  not  mean 
that  all  the  outputs  will  be  correct.  It  only  says  that  all  the  outputs 
which  occur  on  inputs  after  the  correcting  tape,  are  correct. 
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The  use  of  a  random  source  driving  the  automaton  may  or  may  not  be 
appropriate  depending  upon  the  application.  If  we  are  modeling  a  digital 
coder  such  as  might  be  used  to  code  information  on  a  space  satellite 
gathering  radiation  information,  then  it  is  a  good  model.  If,  on  the 
other  hand,  we  are  trying  to  model  a  digital  computing  system  then  it 
is  a  poor  model.  This  is  due  to  the  fact  that  the  input  is  a  program 
which  is  not  random  but  is  truncated.  However,  the  assumption  of  a  ran¬ 
dom  source  at  least  gives  us  a  way  to  start  dealing  with  the  problem. 

It  may  be  possible  in  later  W(  k  to  modify  these  results  for  the  case 
when  we  are  driving  the  automaton  with  a  structured  source. 

1.3  Formal  Languages 

Up  to  the  present  time  the  only  aspect  of  programming  languages 
which  has  been  treated  formally  is  syntax.  The  concept  of  semantics 
or  'meaning'  which  appears  to  be  of  fundamental  importance  both  to  the 
computer  language  designer  and  user  ha6,  up  to  this  point,  not  been 
investigated.  This  omission,  it  appears,  is  due  mainly  to  the  fact 
that  there  does  not  exist,  in  general,  a  formal  definition  of  the  con¬ 
cept.  This  shortcoming  must  be  overcome  before  the  theory  of  language 
can  hope  to  answer  a  number  of  important  questions  arising  in  computer 
languages . 

The  general  theory  of  formal  languages  classifies  languages  into 
several  types  defined  by  the  grammars  which  specify  them.  Chomsky  [b6  ] 
gives  the  types  numeric  designations,  with  type  0  being  the  least  struc¬ 
tured  and  type  3  the  most  structured.  Two  types  have  been  shown  to  be 
equivalent  to  known  classical  structures;  type  0  to  Turing  machines,  and 
type  3  to  finite  state  machines.  Types  1  and  2  are  known  as  context- 
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sensitive  and  context-free  languages,  respectively.  The  most  useful 
representation  in  the  case  of  programming  languages  appears  to  be  con¬ 
text-free  languages.  The  theory  of  context-free  languages,  their  charao- 
terization  and  properties  have  been  thoroughly  studied  [  b6  ] .  In  recent 
years  there  has  been  an  effort  made  to  study  programming  languages  within 
the  framework  of  context-free  languages. 

Any  application  of  the  theory  of  formal  languages  to  programming 
languages  must  differentiate  between  the  reference  language  and  its  real¬ 
ization  on  a  computer,  or  its  hardware  representation.  In  order  to 
define  these  two  we  will  quote  from  the  ALGOL  60  report  [b30]: 

Reference  Language 

(1)  It  is  the  defining  language. 

(2)  It  is  the  basic  reference  and  guide  for  compiler 
builders . 

(3)  It  is  the  guide  for  all  hardware  representations. 

Hardware  Representation 

(1)  Each  one  of  these  is  a  condensation  of  the  reference 
language  enforced  by  the  limited  number  of  characters 
on  standard  input  equipment . 

(2)  Each  one  of  these  uses  the  character  set  of  a  partic¬ 
ular  computer  and  is  the  language  accepted  by  a  trans¬ 
lator  for  that  computer. 

The  reference  languages  of  the  common  problem  oriented  languages 
such  as  ALGOL,  MAD,  and  FORTRAN,  are  context-free  [bl6].  This  property 
only  implies  that  there  exists  a  set  of  rules  or  specifications  defining 
the  syntax  of  the  individual  strings  in  a  problem  oriented  reference 
language  which  are  context-free,  lhis  set  of  rules  along  with  the  set 
of  symbols  is  a  grammar  for  the  reference  language.  Given  this  grammar 
we  know  that  the  process  of  determining  whether  a  given  string  is  an 
element  of  the  language  or  not  is  well  defined.  The  theories  of  automata 
and  context-free  languages  provide  us  with  a  specification  of  a  machine 
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to  implement  this  process.  However  any  attempt  to  generalize  this  notion 
of  membership  to  meaningful  sequences  of  strings  within  the  language  is 
doomed  to  failure  at  present  because  semantics  has  not  been  brought  in. 
Thus  the  theory  gives  us  a  way  to  check  if  the  individual  statements  of 
the  language  are  well  formed  but  at  present  does  not  allow  us  to  determine 
if  a  program  is  meaningful]  that  is,  if  it  will  execute  with  a  given  set 
of  data. 

A  second  area  in  which  the  theory  of  context-free  languages  has  made 
a  contribution  to  programming  languages  is  ambiguity.  The  concept  of  am¬ 
biguity  in  the  formal  theory  is  associated  with  grammars.  Intuitively, 
a  grammar  is  syntactically  ambiguous  if  it  generates  the  same  string  in 
at  least  two  distinct  ways.  This  concept  can  then  be  generalized  to 
languages.  A  language  is  inherently  ambiguous  if  every  grammar  genera¬ 
ting  it  is  ambiguous  [  bl5l  •  It  is  obvious  that  inherently  ambiguous 
programming  languages  are  to  be  avoided.  However  it  is  undecidable 
whether  an  arbitrary  language  is  inherently  ambiguous.  This  should  stim¬ 
ulate  attempts  to  define  a  large  class  of  languages,  rich  enough  for 
programming  languages,  in  which  there  exist  a  decision  procedure  for 
inherent  ambiguity. 

The  formal  definition  of  ambiguity,  unfortunately,  does  not  encom¬ 
pass  all  the  characteristics  of  what  is,  in  general,  meant  by  the  term 
ambiguity  in  programming  languages.  The  missing  element  again  is  seman¬ 
tics.  What  is  needed  is  a  way  to  determine  whether  or  not  a  statement 
'means  '  two  different  things,  for  if  a  statement  means  two  different 
things  the  translating  program  will  have  to  pick  one  and  hence  in  some 
situations  gives  rise  to  a  programming  error.  Within  the  existing  for¬ 
mal  framework  a  detailed  examination  of  to  what  extent  ambiguity  causes 
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programming  errors  is  needed.  This  would  allow  a  characterization  of 
types  of  ambiguity  within  a  programming  language. 

We  next  discuss  the  application  of  the  theory  of  formal  languages 
to  the  translation  problem.  In  formal  languages  the  translation  problem 
is  generally  stated  in  this  form:  Given  a  language  L^  of  type  X  and  a 
language  Lg  of  type  Y,  is  there  a  mapping  h  of  type  Z  such  that  h  maps 
into  (onto,  into  infinite  subset  of)  Lg  [  bl?]  ? 

Unfortunately,  in  practice  a  systems  programmer  is  not  interested 
in  the  existence  of  an  arbitrary  mapping.  He  is  interested  in  the  exist¬ 
ence  of  a  mapping  which  preserves  the  meaning.  In  other  words,  if 
someone  writes  an  ALGOL  program  he  would  like  it  to  be  translated  into 
a  machine  language  program  which  does  what  he  wants  it  to  do.  Thus 
once  again  the  question  of  semantics  arises. 

Programming  language  designers  are  primarily  interested  in  the 
answer  to  the  following  question:  For  a  given  language  L_  (viz.  a 
machine  language)  and  a  give.,  language  L^- (viz.  the  ALGOL  reference  lang¬ 
uage)  can  we  find  a  language  L^  which,  in  some  sense,  is  like  Lg  such 
that  there  is  a  "good"  algorithm  for  translating  L^  into  L^  which  pre¬ 
serves  meaning. 

Work  toward  answering  this  question  has  Just  begun.  So  far  no  work 
has  been  done  on  translation  which  preserves  meaning  or  on  approximating 
one  language  with  another.  However  there  are  now  many  people  working 
on  the  question  of  gocd  algorithms.  The  work  by  Hartmanis  and  Stearns 
is  a  good  example  [b2l].  In  their  paper,  they  give  a  classification  of 
functions  according  to  their  computational  complexity.  The  computational 
complexity  is  a  measure  of  the  speed  with  which  the  function  can  be  cal¬ 
culated  by  a  multi-tape  Turing  machine.  Since  there  is  a  strong  corres¬ 
pondence  between  algorithms  and  multi-tape  Turing  machines,  we  then  have 
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a  measure  of  for  algorithms. 

Fortunately,  the  study  of  formal  languages  has  aided  the  systems  pro¬ 
grammer.  For  one  thing,  by  studying  the  context-free  languages  and  then 
translation,  the  programmer  gets  a  much  better  feeling  of  what  the  trans¬ 
lation  process  is  like  and  what  it  should  do.  For  instance,  by  the  study 
of  push-down  automata,  programmers  have  developed  a  whole  new  technique 
known  as  the  push-down  list.  This  is  of  great  help  in  many  translation 
schemes  as  well  as  for  other  programming  purposes.  The  first  mention  of 
a  scheme  of  this  type  appeared  in  1954  in  a  paper  by  Burks,  Warren,  and 
Wright  [ b4  ] .  For  a  more  current  theoretical  discussion,  see  the  paper 
by  Schutzenberger  [b34],  and  for  a  discussion  of  its  use  in  various  appli¬ 
cations,  see  Evey  [bl2  ]  • 

Schutzenberger  formally  defines  a  special  type  of  automaton  which 
is  not  as  powerful  as  a  Turing  machine  but  has  more  power  than  a  finite 
automaton.  He  calls  this  automaton  a  push-down  automaton  although  his 
use  of  this  term  is  not  exactly  the  same  as  that  of  other  authors.  He 
then  shows  that  the  set  of  words  recognized  by  these  automata  are  unam¬ 
biguous  context-free  languages  and  that  there  is  a  weak  converse  to  this 
property.  Evsy's  methods,  on  the  other  hand,  are  much  less  formal.  He 
defines  a  class  of  machines  called  push-down  machines.  Then  he  shows  that 
for  every  context-free  language  there  corresponds  a  push-down  machine 
that  recognizes  it,  and  conversely,  the  set  of  strings  recognized  by 
any  rush-down  machine  is  a  context-free  language. 

This  brings  us  to  a  consideration  of  the  hardware  representation. 

In  general,  for  a  given  reference  language,  there  will  be  many  compiler 
realizations  of  the  lang-  age.  Furthermore,  due  to  the  physical  consider¬ 
ations  of  finiteness,  each  of  the  compiler  languages  is  only  an  approximate 
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realization  of  the  reference  language.  These  approximations,  in  general, 
amount  to  a  truncation,  thus  giving  a  finite  language.  However  the  full 
extent  of  the  restrictions  that,  a  compiler  places  on  a  reference  language 
is  not  known  in  general.  It  is  an  open  question  to  what  degree  the  theory 
of  pontext-free  languages  will  be  useful  for  compiler  languages.  Further 
Investigation  along  these  lines  is  certainly  called  for. 

In  summary,  then,  the  situation  is  as  follows.  The  theory  of  formal 
languages  today  is  only  of  limited  use  to  someone  writing  compiler 
languages.  As  the  theory  is  extended  to  more  class  of  languages,  par¬ 
ticularly  those  intermediate  between  context-free  languages  and  finite 
automata  languages,  it  will  become  more  and  more  useful.  Also,  if  the 
syntactic  theory  were  extended  to  include  some  of  the  rudimentary  concepts 
from  semantic  theory,  there  would  be  even  more  areas  of  potential  appli¬ 


cation. 
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2.  Theory  of  Algorithms 
2.0  Introduction 

The  theory  of  algorithms  relates  to  computer  programs  in  the  same  way 
automata  theory  relates  to  the  structure  and  the  behavior  of  computers. 

(We  interpret  the  word  algorithms  in  a  broad  sense,  without  limiting  our 
attention  to  formal  systems  of  the  Markov  type.)  The  theory  of  algo¬ 
rithms  encompasses  such  questions  as:  How  should  algorithms  be  formally 
characterized?  What  representations  are  appropriate?  How  do  the  compu¬ 
tational  complexity  and  storage  requirements  of  a  given  algorithm  vary 

with  the  problems  in  its  domain?  Does  a  given  algorithm  terminate  for 

<# 

all  problems  in  its  domain?  Is  the  result  of  applying  an  algorithm 

t 

unique  for  ail  problems  in  its  domain?  When  does  an  optimal  algorithm 
exist  (for  some  reasonable  definition  of  optimality)?  How  can  an  optimal 
algorithm  be  determined? 

Our  program  in  this  urea  encompasses  three  projects: 

(1)  The  study  of  "monotone  congruence"  algorithms  by  ft.  F. 
Arnold  and  D.  L.  Richards.  A  technical  report  has 
been  issued  and  journal  publication  is  expected. 

(2)  The  formalization  of  discrete  dynamic  programming  algo¬ 
rithms,  and  the  investigation  of  its  connections  with 
automata  theory,  by  R.  M.  Karp. 

(3)  The  study  of  "branch -and-bound"  algorithms^  being 
carried  on  by  E.  L.  Lawler  and  D.  E.  Wood,  under  principal 
sponsorship  of  National  Science  Foundation  Grant  GP-277S. 
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2.1  Monotone  Congruence  Algorithms 

Given  an  associative  system  (semi-group)  in  which  each  element  is 
assigned  a  cost  and  in  which  an  equivalence  relation  obtains  between 
elements,  it  is  often  of  interest  to  ask  the  question:  What  is  the  least 
costly  word  equivalent  to  a  given  word?  Three  examples  of  such  problems 
are  the  travelling  salesman  problem  studied  by  Dantzig,  et  al.  [b8  ], 
the  optimum  control  problem  (cf.  Pontryagj  ib33])>  and  the  problem  of 
finding  the  least  word  which  performs  a  giver,  mapping  upon  the  states  of 
a  finite  automaton.  Algorithms  do  nc-t  exist  for  all  such  problems,  since 
the  solution  may,  in  general,  require  the  solution  of  the  word  problem 
for  semi -groups,  which  is  known  to  be  unsolvable  (Davis  [b9]). 

The  problems  mentioned  above  involve  natural  two-sided  congruence 
relations.  Two  input  words  x  and  y  to  a  finite  automaton  may  be  regarded 
as  equivalent  if  they  perform  the  same  mapping  from  the  set  of  states  of 
the  automaton  to  itself;  i.e.,  if  M(s,x)  =  M(s,y)  for  all  states  s  where 
M  is  the  transition  function  of  the  automaton.  A  simplified  model  of 
computer  programming  can  be  obtained  by  reinterpreting  the  input  words  as 
programs,  and  the  initial  state  as  data  upon  which  the  j  .-grams  act;  here 
the  programs  are  not  self-modifying  and  contain  no  instructions  which 
transfer  control.  Programs  P  and  Q  are  then  equivalent  if  they  reach  the 
same  result  for  each  set  of  data;  in  that  case  the  programs  RPS  and  RQS 
must  also  be  equivalent,  R  being  run  immediately  before  P  (or  Q)  and  S 
being  run  immediately  after.  Thus  the  equivalence  relation  satisfies 
the  definition  of  a  two-sided  congruence. 

The  programming  and  automaton  problems  can  be  solved  by  enumeration 
if  necessary;  however,  the  solution  algorithms  are  often  impractical  and 
may  give  no  intuitive  insight  into  the  structure  of  the  optimizing  process 
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itself.  It  is  hoped  that  greater  insight  and  more  effective  algorithms 
can  he  found  by  studying  the  general  class  of  associative  systems  with 
two-sided  congruence  relations,  a  class  which  contains  problems  of  all 
degrees  of  difficulty. 

The  theory  of  monotone  congruence  algorithms  applies  if  a  two- 
sided  congruence  relation  exists  and  there  exists  a  total  order  relation  , 
on  the  costs  of  the  words,  which  has -.certain  natural  properties.  The 
properties  of  this  class  of  algorithms  has  been  investigated  [  a2  ]  and 
a  general  result  concerning  order  relations  on  finite  alphabets  estab¬ 
lished. 

A  Markov  normal  algorithm  on  a  set  of  words  £*  is  a  finite  ordered 
list  of  substitutions  of  2*.  For  an  order  relation  and  an  equivalence 
relation,  a  monotone  congruence  algorithm  is  a  Markov  normal  algorithm 
A  for  which: 

(1)  A(w)  is  minimal  whenever  it  exists. 

(2)  for  each  substitution  x  -*  y  of  A,  x  >  y  and  x  ~  y. 

In  terms  of  computer  programming,  the  words  may  be  programs  or 
systems  of  subroutines,  the  letters  of  the  words,  instructions  or  sub¬ 
routines.  The  order  relation  relates  the  cost  of  a  program,  i.e.  in  terms 
of  execution  time,  space  required,  etc.  The  equivalence  relation  equates 
programs  which  perform  the  same  action  on  the  internal  states  of  the  com¬ 
puter.  A  monotone  congruence  algorithm  is  then  a  succession  of  replace¬ 
ments  of  a  subprogram  with  a  less  costly  one  which  computes  the  same 
function. 

The  first  principal  result  established  is  that  for  any  monotone 
congruence  algorithm  A  and  any  word  w,  A(w)  exists;  i.e.,  that  every 
monotone  congruence  algorithm  always  terminates.  The  proof  depends  on 
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a  quite  general  theorem  about  total  order  relations  on  £*,  i.e.: 

Every  total  order  which  is  a  refinement  of  the  inclusion  partial 
ordering  is  a  well-ordering. 

Auxiliary  letters  are  parts  of  words  which  appear  only  in  nhe  inter¬ 
mediate  stages  of  the  application  of  an  algorithm.  Concluding  substitu¬ 
tions  of  an  algorithm  are  those  which  terminate  the  process  immediately, 
without  exhausting  all  possible  other  steps.  It  is  proved  in  [  a2  ]  that 
every  monotone  congruence  algorithm  is  equivalent  to  one  which  has  neither 
auxiliary  letters  nor  concluding  substitutions. 

The  second  principal  result  is  that  every  class  of  mutually  equivalent 
monotone  congruence  algorithms  contains  a  unique  minimal  algorithm,  called 
the  core  algorithm,  which  can  be  obtained  by  applying  any  member  of  the 
class  to  itself.  The  algorithm  is  unique  up  to  the  order  of  the  substi¬ 
tutions  and  minimal  with  respect  to  the  number  of  substitutions.  A 
uniformly  optimal  algorithm  is  one  which  requires  the  fewest  number  of 
steps,  or  applications  of  the  algorithm,  to  terminate.  The  study  of 
optimality  reduces  to  the  study  of  the  algorithms  which  can  be  produced  by 
reordering  the  substitutions  of  the  core  algorithm.  It  has  -sen  shown, 
however,  that  a  uniformly  optimal  ordering  of  the  substi\  .  ons  oes  not 
always  exist,  and  that  more  than  one  uniformly  optimal  ordering  may  exist. 

2.2  Discrete  Dynamic  Programming 

Over  the  paBt  several  years  dynamic  programming  has  emerged  as  an 
important  computational  tool  for  the  solution  of  multistage  optimization 
problems.  It  is,  therefore,  surprising  that  the  technique  has  never  been 
precisely  defined,  and  that  the  Principle  of  Optimality,  which  underlies 
dynamic  programming,  has  not  been  given  a  precise  and  general  statement. 


R.  M.  Karp,  together  with  M.  Held  of  International  Business  Machines, 
Incorporated,  has  undertaken  to  construct  a  mathematical  theory  encom¬ 
passing  the  domain  of  discrete,  deterministic  dynamic  programming. 

The  two  central  concepts  of  this  theory  are  the  discrete  decision 
process  and  the  sequential  decision  process.  A  discrete  decision  process 

D  =  (A,S,P,f) 

is  specified  by  A,  a  finite  set  of  primitive  decisions,  S,  a  subset  of 
all  finite  sequences  of  decisions,  P,  a  space  of  parameters  (i.e.,  prob¬ 
lem  data)  and  a  real -valued  cost  function  f  with  domain  S  X  P.  The 
elements  of  S  represent  feasible  policies,  and  the  problem  is  to  construct 
an  algorithm  to  find,  for  any  given  p€P,  an  element  seS  minimizing  f(s,p). 
A  sequential  decision  process  is  specified  by  a  finite  automaton  Q/ 
with  input  alphabet  A  which  recognizes  the  "event"  S,  a  parameter  space 
P,  and  a  function  h  with  the  following  interpretation:  if  the  parameter 
specification  is  p,  and  state  q  of  CL  has  been  reached  by  an  input  se¬ 
quence  x  having  cost  |,  then  the  cumulative  cost  after  the  further  appli¬ 
cation  of  input's  is  h(|,q,a,p).  The  process  is  monotone  if  h  is  an  in¬ 
creasing  function  of  £. 

Thus,  a  sequential  decision  process  consists  of  a  finite  automaton 
together  with  additional  cost  structure.  The  minimization  problem  for 
a  sequential  decision  process  is  that  of  finding  an  input  sequence  in 
the  set  S  having  minimum  cost.  A  principal  result  of  the  theory  is  that 
if  the  sequential  decision  process  is  monotone,  this  minimization  prob¬ 
lem  reduces  to  the  solution  of  a  system  or  recurrence  equations  of  the 
type  associated  with  dynamic  programming;  and,  conversely,  every  such 
system  of  recurrence  equations  can  be  associated  with  the  minimization 
problem  for  some  monotone  sequential  decision  process.  A  second  principal 
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result  is  the  characterization  of  the  possible  representations  of  a  dis¬ 
crete  decision  process  by  a  monotone  sequential  decision  process  having 
the  same  minimization  problem.  This  problem  is  akin  to  that  of  relating 
behavior  to  structure  in  finite  automata,  and  similar  methods  are  used. 

Thus,  we  feel  that  a  useful  and  novel  coupling  of  automata  theory 
with  optimization  theory  has  been  achieved.  It  is  our  conviction,  based 
on  the  examination  of  many  examples,  that  any  dynamic  programming  formu¬ 
lation  of  the  optimization  problem  for  a  discrete,  deterministic  process 
can  faithfully  be  interpreted  as  the  representation  of  that  process  by  a 
monotone  sequential  decision  process. 

2.3  Branch-and-Bound  Algorithms 

.Among  the  most  general  and  most  useful  approaches  to  the  solution  of 
constrained  optimization  problems  is  that  of  "branching-and-bounding"  [  al4] 
(or,  according  to  Bertier  and  Boy  [b2  ],  "separation  et  evaluation 
progressives").  Most  commonly,  this  is  a  technique  under  which  the 
space  of  feasible  solution  is  repeatedly  partitioned  into  smaller  and 
smaller  subsets,  and  a  lower  bound  (in  the  case  of  minimization)  is 
calculated  on  the  cost  of  the  solutions  within  each  subset.  When  the 
bound  for  any  subset  exceeds  the  cost  of  a  known  feasible  solution,  it 
follows  that  no  solution  within  the  subset  can  be  optimal.  The  parti¬ 
tioning  continues  until  a  feasible  solution  is  found  such  that  its  cost 
is  no  greater  than  the  bound  for  any  subset. 

There  has  been  some  recent  interest  in  branch- and -bound  methods 
for  integer  linear  programming  and  for  the  traveling  salesman  problem. 

These  algorithms  should  perhaps  be  characterized  as  "meta-algorithms,", 
since  they  represent  methods  by  which  known  algorithms  can  be  embedded 
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subject  to 
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gp^x)  >  0 


g^(x)  >  0 


(J) 


and  xeX^ . 

A  definition  of  the  branch -and-bound  process  can  be  given  induc¬ 
tively.  That  is,  given  a  set  of  bounding  problems  at  some  intermediate 
point  in  the  calculations,  it  is  specified  how  a  new  set  of  bounding 
problems  is  obtained  by  "branching". 

We  let  the  current  set  of  bounding  problems  be  denoted  (l),  (2), 
(3),  . ..,  (p) ■  In  order  to  be  a  valid  set  of  bounding  problems,  it  is 
sufficient  that  the  following  bounding  requirement  be  satisfied: 

(b)  If  x  is  a  feasible  solution  to  problem  (0),  then  there  exists 
a  bounding  problem  (j)  for  which  x  is  feasible  and  c^(x)  <  c^(x). 

The  bounding  problems  are,  by  assumption,  "easy"  problems,  and  can 
be  solved  by  appropriate  direct  methods.  Let  the  optimal  solutions 

thereby  obtained  be  denoted  x  x^,  ...,  x^,  respectively.  The 

(k) 

solution  x'  '  is  optimal  for  problem  (o)  if 

(51)  c^(x^)  <  c^(x^),  for  j=l,2,...,p 

(52)  x^k)  is  feasible  for  problem  (0) 

(53)  c(°)(*W)  =  cW(xW). 

A  solution  x^  which  satisfies  (Si),  (S2),  and  (S3)  solves  the 
original  problems.  "Branching",  i.e.  replacing  one  of  the  existing 
bounding  problems  by  two  or  more  new  bounding  problems,  must  continue 
until  a  solution  which  satisfies  (Si),  (S2),  and  (S3)  is  found. 

The  work  under  this  project  has  encompassed: 

(l)  A  survey  of  branch -and -bound  algorithms  as  applied  to  various 


types  of  optimization  problems. 

The  development  of  new  branch-and-bound  algorithms 
for  problems  of  interest. 

An  investigation  of  optimal  strategies  *  ,r  branching. 
I.e.,  strategies  which  minimize  the  lent  ..  ^f  compu¬ 
tation  or  the  amount  of  computer  storage. 

The  development  of  a  stochastic  model  for  branching 
algorithms.  It  is  hoped  that  this  model  can  be  used 
to  predict  when  branch-and-bound  methods  can  be  success¬ 
fully  applied  and  when  they  cannot:  i.e.,  which  distri¬ 
butions  of  costs  in  the  domain  of  feasible  solutions  are 
f aver able. 


and  which  are  not. 
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3 .  Systems  Analysis 
3.0  Introduction 

The  developnent  of  models  for  the  analysis  and  study  of  the  func¬ 
tional  parts  of  information  processing  systems  and  the  classification 
of  information  processing  systems  and  problems  are  the  objectives  of 
this  part  of  the  research  program.  The  model  of  a  functional  part 
of  an  information  processing  system  should  permit  comparisons  to  be 
made  between  known  solutions  and  should  suggest  new  solutions.  This 
research  is  abstract  but  practical  problems  in  information  processing 
systems  design  have  a  direct  influence  on  the  formulation  of  the 
models  and  the  questions  asked. 

Major  emphasis  is  given  to  algebraic  models  though  statistical 
models  and  simulation  are  not  excluded  from  the  study.  Automata  theory 
models  are  used  when  applicable.  Models  have  been  developed  to  study 
problems  associated  with  storage,  multiprocessor  assignment,  and  control. 
R.  F.  Arnold  has  developed  a  model  for  the  study  of  the  addressing  for¬ 
mat  for  random  access  storage.  Finite  state  machines  are  used  in  the 
formulation  but  statistical  techniques  are  used  in  the  evaluation.  R. 
Jump  has  simulated  the  dynamic  allocation  of  storage.  Karp  and  Reiter 
[al2,al3]  have  extended  and  generalized  a  graph  model  of  concurrent  com¬ 
putation  due  to  Karp  and  Miller  [b24]. 

Control  models  emphasizing  topology  and  quantity  of  control  infor¬ 
mation  have  been  formulated  by  H.  L.  Garner  and  have  shown  the  need 
for  a  more  definitive  relationship  between  the  rate  of  computstioh  and 
the  rate  of  control  information,  which  is  now  being  studied.  Classifi¬ 
cation  and  complexity  studies  are  still  in  the  formulative  stage. 
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3.1  Storage  Format  Characterization  for  Random  Access  Storage  j 

The  large  random  access  store  or  file  is  an  important  part  of  j 

existing  computer  systems  and  will  continue  to  be  important  in  the  -  j 

future  because  of  the  continuing  requirement  for  large  quantities  of  j 

I 

reference  data.  Practical  problems  exist  with  respect  to  the  efficient  j 

utilization  of  such  a  store.  In  general,  it  is  desirable  to  minimize  J 

the  number  of  accesses  to  the  store  required  to  obtain  a  given  record  ; 

stored  in  the  file  and  to  maximize  the  number  of  filed  words  in  the 
store.  It  is  not  feasible,  in  general,  to  set  up  a  one-to-one  corres¬ 
pondence  between  A,  the  set  of  hardware  addresses  of  the  store  and  the 

K,  the  set  of  keys  which  are  the  identifying  part  of  each  record.  The 

difficulty  stems  from  the  fact  that  not  all  keys  are  active  and  the  j 

active  keys  are  not  usually  distributed  uniformly  ever  the  key  set.  A 

sorted  file  provides  a  poor  solution  to  this  problem  since  a  file  sort 

Is  required  whenever  new  records  are  added.  The  open  file  concept  pro¬ 
vides  a  reasonable  solution.  In  a  general  open  file  system  there  exists 
a  deterministic  algorithm  which  generates  a  sequence  of  store  addresses 
for  each  key.  The  record  is  stored  in  the  first  empty  storage  location 

V 

associated  with  the  set  of  addresses  generated  by  the  algorithm.  The  j 

algorithm  computes  f ,  sometimes  called  a  hash  function,  which  maps  K  in-  I 

| 

to  A.  Thus  f:  K  -»  A.  In  general,  f  is  a  many-to-or.e  mapping.  Ideally,  * 

f  should  map  the  same  number  of  keys  to  each  element  of  A  but  adjacent  j 

keys  should  be  mapped  into  separate  addresses  in  order  to  break  up  clus¬ 
ters  of  active  keys.  An  error  correcting  group  code  has  the  property 
that  any  code  word  is  at  least  a  distance  d  from  any  other  code  word.  A 
decoding  function  g  maps  the  set  of  all  possible  received  code  words 
onto  the  set  of  messages;  g:  R  -*  M.  Two  elements  of  R  which  map  onto 
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the  same  element  of  M  are  separated  by  at  least  a  distance  d  and  an  equal 
number  of  elements  of  R  are  associated  with  each  element  of  M.  Thus  g 
is  a  useful  hash  function  and  the  theory  of  codes  provides  the  informa¬ 
tion  required  to  design  this  type  of  hash  function.  Number  system  con¬ 
version  algorithms  have  also  been  considered  for  the  generation  of  hash 
functions.  Hash  addressing  has  been  effectively  applied  [bl  ]  to  obtain 
a  store  organization  for  list  processing.  Other  applications  appear 
possible  but  these  have  not  been  studied  in  detail. 

A  general  characterization  of  the  storage  format  problem  for  random 
access  stores  has  been  obtained  by  R.  F.  Arnold  and  some  examples  anal¬ 
yzed.  In  the  type  of  system  considered,  there  is  a  conventional  random 
access  store  and  a  finite  automaton  which  plays  the  role  of  "reader- 
interpretor" . 

A  random  access  store  is  given  by  a  pair  of  sets  A  and  W. 

The  "state"  of  the  store  is  given  by  an  arbitrary  function  c:  A  -»  W 
termed  the  contents  function. 

A  format  for  the  representation  of  functions  from  the  set  X  into 
the  set  Y  is  given  by  a  finite  automaton  with  the  state  set  S,  alphabet 
W  and  transition  function  M:  S  x  W  -»  S.  The  "output"  of  the  automaton 
is  given  by  a  map  <p:  S  -*  A  U  Y.  Also  defined  is  an  "initial  state" 
function  \|n  X  -*  S . 

A  format  determines  a  mapping  from  the  states  cf  the  store  (A,W) 

into  the  set  of  partially  defined  functions  from  X  into  Y.  Let  c  be 

a  contents  function:  then  f  :  X  -»  Y  is  defined  as  follows.  We  first 
’  c 

define  a  partially  defined  function  \  from  S  into  Y. 

If  \[r( s)  e  Y  then  \  (s)  =  Cp(s). 

If  cp(s)  ^  Y  then  \c(s)  =  Xc(M(s,c(q>(s)))) . 
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If  ^c(s)  is  not  determined  by  either  of  the  above,  then  it  is  left 
undefined. 

Then  fc(x)d=f\c(\j/(x))  • 

In  a  sense,  this  is  similar  to  the  repeated  use  of  hash  functions. 
Here,  if  the  output  of  the  automaton  is  not  in  the  range  of  the  function, 
the  transition  function  is  applied  to  obtain  a  new  state  when  e  output  may 
be . 

One  initial  problem  is  that  of  determining  for  some  given  function 
from  X  to  Y  and  a  given  format,  a  contents  function  which  "represents" 
it.  Practical  application  requires  that  the  representative  be  easily 
computed  and  that  given  a  representation  of  one  function,  a  representative 
of  a  "neighboring"  function  be  very  easily  computed.  By  a  neighbor  here 
we  mean  a  function  whose  values  differ  for  only  a  few  arguments.  Also  of 
interest  is  the  obtaining  of  useful  characterizations  of  the  class  of  func 
tions  represented  by  a  given  format. 

This  section  gives  examples  of  certain  formats  where  very  satisfac¬ 
tory  information  can  be  obtained.  All  of  these  have  the  property  that 
the  structure  of  the  automaton  is  pseudo-probabilistic ally  determined 
which  then  justifies  certain  probabilistic  arguments  as  to  the  existence 
of  representatives  and  the  computational  and  information  theoretic 
efficiency  of  the  formats. 

The  automata  in  general  "contain"  hash  functions.  In  fact,  this 
entire  work  can  be  considered  as  a  certain  natural  extension  of  the 
hash  function  technique  in  which  the  primary  disadvantage  of  hash 
addressing,  namely  the  many-to-one  character  of  the  hash  function,  is 
obviated  by  artificially  enlarging  the  range  of  the  hash  functions. 

The  work  to  date  appears  to  provide  significant  improvements  in 
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handling  a  wide  variety  of  content-addressability  problems  where  the  read 
to  write  ratio  is  fairly  high. 

Of  more  abstract  interest  is  the  relation  between  the  structure  of 
the  automaton  and  the  difficulty  of  the  optimum  representative  computa¬ 
tion.  For  certain  formats  this  computation  is  trivial  while  for  others 
it  appears  to  become  arbitrarily  difficult. 

We  now  consider  three  examples  which  are  specific  realizations  of 
the  general  formulation  for  the  random  access  store. 

The  linear  functional  format  is  the  most  efficient  with  respect  to 
storage  space  at  the  expense  of  requiring  extensive  computations  to  com¬ 
pute  the  appropriate  representation  of  a  function. 

For  this  example  the  range  of  the  function  Y  =  *  [0,1}  and  a 

store  with  n-bit  words  is  used  so  W  =  (Z2)n.  The  binary  words  of 
storage  are  treated  as  vectors  over  the  mod  2  Held. 

Let  T  be  an  auxiliary  "copy"  of  (Z2)n.  The  hash  functions  employed 

map  fr^m,  X,  the  domain  of  the  function,  into  the  product  set  AxT, 

where  A  is  the  set  of  store  addresses.  For  the  i-th  hash  function 

^(x)  »  (ai(x),t1(x))  1=1,2,... 

The  contents  for  c  assigns  a  particular  element  of  W  to  each  etore 

address  acA;  c(a)  »  v«W.  Assume  c  has  been  determined.  If  c(aj)  /  0 

and  c(a. )  ■  0  i  <  J,  then  f  (x)  »  c(a. (x)) ‘t, (x)  (dot  product) 

l  c  j  j 

-  wtj(x)  «  l2. 

If  cCa^)  -  0  for  all  i,  then  f(x)  is  undefined. 

For  a  given  function  f :  X  -♦  Y,  we  now  Consider  the  process  by  which 
a  representative  c  is  computed,  if  it  exists. 

Consider  the  subsets  in  X  which  are  the  inverse  images  of  all  a 
under  h^.  For  each  location  acA,  either  a  w<W  exists  or  it  doeo  not  such 
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that  w-t(x)  =  f(x)  for  all  x  in  the  inverse  image  of  a.  If  no  such 
w  exists  c(a)  is  set  equal  to  zero,  and  all  elements  in  the  inverse 
image  are  assigned  new  addresses  using  h^.  It  is  now  necessary  to 
recheck,  for  all  x,  fc(x)  =  w<t(x)  since  new  x's  may  have  been  added 
to  the  image  set  of  any  aeA.  This  will  result  in  more  a's  being 
assigned  contents  zero,  and  create  certain  new  "unhoused"  x's.  If 
this  process  terminates,  then  clearly,  a  representative  c  has  been 
found,  while  if  it  does  not,  no  such  c  exists.  This  last  statement 
has  not  been  proven;  however,  t  le  c  derived  by  the  above  process  can 
be  shown  to  be  uniformly  optimal  in  the  sense  that  the  set  of  a's 
whose  contents  is  zero  in  any  representative  c  contains  the  set  of  a's 
set  to  zero  by  the  above  process.  Is  the  probability  that  a  vector 
exists  in  (Z2)n  which  satisfies  m  randomly  selected  linear  constraints 
the  principal  information  necessary  to  quantitatively  analyze  the 
feasibility  of  this  storage  format?  A  lower  bound  for  this  probability 
is  given  by  the  probability  that  m  vectors  in  (Z2)n  are  linearly  inde¬ 
pendent,  which  is  p(m,n)  >  1  -  l/2n~n. 

Observe  that  p(m,n)  is  dependent  only  on  (n-m).  The  number  of 
constraints  that  can  be  applied  to  a  given  location  can  be  increased 
without  a  decrease  in  the  probability  that  the  constraints  can  be  satis¬ 
fied  if  n-m  is  held  constraint.  Thus  a  requirement  for  x  additional 

« 

constraints  require  x  additional  bits  per  word  in  store. 

If  a  representative  c  exists,  then  there  exists  an  average  inverse 
image  size  for  each  address  location.  Furthermore,  if  X  is  large  and 
the  hash  functions  are  well  behaved,  the  probability  distributions  of 
address  inverse  Image  size  will  be  Poisson  with  mean  It  is  then 
possible  to  write  equilibrium  equations  which  must  hold  under  the 

i. 


assumption  that  the  computation  process  terminates . 

Let  p  he  the  proportion  of  locations  set  equal  to  zero. 


P  = 


n-1 


2 


m=n 


An  ideal  hash  function  assigns  X/A  =  p,  to  each  location  a.  We  must 
also  have: 


\ 


P  +  PP  + 


2 

P  P 


M 

1-p 


These  relations  form  the  necessary  basis  for  study  of  the  process. 

The  hash  format  is  closely  related  to  the  conventional  technique 
of  storing  in  each  location  both  the  x  and  f(y)  values.  The  technique 
does  nothing  more  than  replace  x  with  its  hash  image,  which  at  once 
removes  the  structure  from  X  and  in  so  doing  saves  space. 

This  format  is  defined  as  follows.  We  require  an  auxiliary  set 
T,  whose  size  must  be  decided  upon  by  a  consideration  of  other  parameters 
of  the  problem.  The  set  W  is  interpreted  as  (TxY)  U  Q  where  Q  is  a  set 
of  one  or  more  elements  which  are  used  to  indicate  that  the  standard 
format  interpretation  does  not  apply  and  the  next  location  in  the  se¬ 
quence  must  be  examined. 

The  hash  functions  used  map  X  into  A  XT. 

hjOO  »  (e^x),  t^x)). 

Let 


c(ai(x))  /  q  and  t^x)  =  t#^) 

the  #  is  use!  to  indicate  values  stored.  t*(ai)  designates  the  tag  part 
of  the  contents  of  a^,  and  let  i  be  the  least  such  i  for  which  this 


holds .  Then 
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where  y*(a  )  indicates  the  portion  of  the  contents  of  from  the  set  Y. 
If  no  such  i  exists,  then  fc(x)  is  left  undefined. 

More  informally,  the  reading  algorithm  consists  in  looking  in 
successive  locations,  continuing  as  long  as  one  finds  either  a  "q"  or  a 
hash  tag  which  does  not  match. 

A  computation  of  representatives  for  this  process  is  fairly 
straight -forward.  One  "stores"  the  values  one  at  a  time,  keeping  track 
in  a  separate  store,  of  the  set  of  x's  mapping  into  each  address  so  far. 
For  each  xeX,  one  computes  h^(x)  and  if  nothing  yet  has  been  mapped  into 
a1(x),  the  pair  (t*(x),  y(x))  is  stored  there.  If  some  x'eX  has  already 
been  mapped  into  the  locations  resulting  in  (T(x^),  y(x^))  having  been 
stored  there,  compare  t*(x')  with  t(x) .  If  they  are  different,  go  on  to 
h^(x),  repeating  the  above  process.  If  tn (x)  f  t*(x')  then  not  only 
must  one  continue  onto  h2(x)  but  also  the  contents  of  a^Cx)  must  be  set 
equal  to  q  and  x'  must  be  put  back  into  the  set  of  as  yet  unassigned  x's. 
If  a^(x)  already  is  set  to  q,  go  on  to  hg. 

For  a  given  size  store  and  given  X  and  Y,  it  is  of  interest  to  con¬ 
sider  how  the  store  should  be  partitioned  so  as  to  yield  the  fewest  num¬ 
ber  of  seeks  per  reference.  Clearly,  as  T  is  made  large,  A  is  reduced 
resulting  in  more  references  to  each  address.  On  the  other  hand,  each 
reference  that  is  made  is  less  likely  to  result  in  the  match  of  hash 
tags  for  different  x's  and  hence  the  "loss"  of  a  location  as  a  value 
storing  slot. 

In  a  manner  similar  to  that  used  in  the  linear  format,  a  pair  of 
equations  may  be  written  describing  the  equilibrium  state  of  the  writing 
process,  which  must  be  analyzed  numerically  to  obtain  quantitative 
information  about  the  process  and  the  optimum  value  of  the  parameters 
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stored. 

The  multistate  reader  format  technique  is  of  particular  interest 
for  representing  functions  with  highly  unbalanced  distributions  on  their 
ranges. 

For  this  example  let  Y  =  £a,b)  and  W  =  (0,1).  The  hash  functions 
used  map  from  X  to  A. 


Let 

cClbCx))  =  0 

and 

c(h  (x))  =  1 

J  <  i 

Then 

fc(x)  =  a 

iff 

i  =  2  mod  3 

b 

iff 

i  ^  2  mod  3. 

If  cOi^x)) 

=  1  for  all  i, 

fc(x) 

is  not  defined. 

The  process  of  computing  a  representative  for  a  given  function  is 
fairly  easy,  and  closely  resembles  that  used  with  the  linear  format. 

Start  out  with  all  locations  set  equal  to  0.  For  each  xeX  in  turn, 
set  equal  to  1,  those  locations  necessary  for  that  x  to  be  correctly 
mapped.  It  will  be  necessery  to  go  through  the  x's  several  times  until 
one  gets  all  the  way  through  without  making  any  changes  in  c.  The  re¬ 
sulting  c  is  uniformly  best  since  at  each  step  we  only  set  equal  to  1 
those  locations  which  must  be  set  to  1  for  any  representative.  Of 
course ,  this  process  may  not  terminate  in  which  case  no  representation 
exists . 

One  generalization  of  the  above  format  is  to  replace  the  modulo 
three  counter  with  arbitrary  n  >  2,  and  map  the  n  congruence  classes 
arbitrarily  into  some  Y.  Any  suei.  format  will  have  essentially  the 
same  representative  computation  procedure  as  the  above. 

For  large  n  and  all  but  one  congruence  class  mapped  into  the  same 
element  y*eY,  it  is  clear  that  if  c  were  chosen  at  random,  the  probability 
that  fc(x)»y*  is  very  large.  Hence,  "mo6t"  of  the  contents  functions 


i 
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represent  functions  which  are  highly  unbalanced  on  their  range.  While 
this  is  not  in  itself  a  proof  that  such  a  format  is  appropriate  for 
representing  such  functions,  it  is  a  necessary  condition  and  hints  at 
how  such  a  proof  might  go. 

Still  further  generalization  of  the  above  format  suggests  replacing 
the  3  counter  with  an  arbitrary  finite  automaton.  In  one  such  generali¬ 
zation  the  automaton  is  a  5-state  machine  in  which  two  of  the  states 
are  terminal  and  have  output  labels  from  Y .  Observe  that  events  are 
associated  with  each  element  of  Y 


E  =  (111)*  HO 

a 

=  (111)*  (0  U  10) 

which  represent  the  first  arrivals  to  states  having  the  respective  labels 
In  general,  any  finite  automaton  having  input  alphabet  W  and  sane  of  its 
states  labelled  with  elements  of  Y,  serves  to  define  a  format. 

Additional  details  and  extc-nsions  can  be  found  in  F.  F.  Arnold  [al]. 
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3.2  Dynamic  Allocation  of  Storage 

A  set  of  procedures  for  dynamic  allocation  of  storage  known  to 
have  been  considered  for  implementation  in  an  important  commercial 
system,  were  chosen  for  evaluation.  By  means  of  simulation,  it  was 
found  that  the  procedures  were  not  significantly  different  in  their 
performance;  hence,  the  most  easily  implemented  procedure  within  the 
class  should  be  chosen. 

It  was  assumed  that  the  memory  is  divided  into  pages,  and  that  the 
pages  are  assigned  to,  and  released  from,  programs  as  needed.  Each 
program  requests  and  releases  small  blocks  of  contiguous  words  within 
those  pages  assigned  to  it.  Blocks  being  used  by  a  program  ere  called 
active  blocks,  and  those  blocks  that  are  not  being  used  but  which  are 
within  one  of  the  pages  assigned  to  a  program  are  called  inactive  blocks. 

When  a  page  is  first  assigned  to  a  program,  it  is  considered  to  be 
a  single  inactive  block.  As  t.he  program  uses  it,  it  becomes  broken  into 
smaller  blocks,  both  active  and  inactive.  New  active  blocks  are  created 
from  within  old  inactive  blocks.  If  there  is  no  inactive  block  within 
any  of  the  pages  assigned  to  the  program  which  is  large  enough  to  satisfy 
a  request  for  a  new  active  block,  a  new  page  is  assigned  to  the  program. 
When  a  block  becomes  inactive,  it  combines  with  inactive  blocks  on  either 
side  to  form  a  single  inactive  block.  When  the  last  (highest  numbered) 
page  to  be  assigned  to  the  program  becomes  inactive,  it  is  released 
from  the  program.  The  object  of  the  allocation  procedures  is  to  minimize 
the  average  number  of  pages  assigned  to  the  program. 

There  an  two  somewhat  incompatible  heuristics  that  can  be  invoked. 
One  is  to  fill  each  new  request  with  the  smallest  inactive  block  larger 
than  the  request,  without  regard  to  the  page  in  which  this  block  is 
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The  reasoning  here  is  that  the  larger  inactive  blocks  should  be  con¬ 
served,  in  order  to  reduce  the  probability  that  a  future  request  can¬ 
not  be  filled  vith  any  existing  inactive  block.  The  other  heuristic 
is  to  fill  each  request  with  the  first  (lowest  numbered)  available  in¬ 
active  block  larger  than  the  request,  without  regard  to  its  size.  The 
reasoning  here  is  that  one  should  use  the  lowest  numbered  blocks,  in 
order  to  increase  the  probability  that  the  higher-numbered  pages  can 
be  released  in  the  near  future. 

A  compromise  between  these  two  heuristics  is  obtained  by  separating 
the  pages  into  two  sets.  The  lower -numbered  pages  are  assigned  to  the 
first  set,  and  the  "smallest  block"  heuristic  is  applied  to  them.  If 
a  request  cannot  be  filled  within  the  first  set,  an  attempt  is  made  to 
fill  it  from  within  the  second  set,  using  the  "first  block"  heuristic. 
The  pages  can  be  separated  into  the  two  sets  according  to  a  fixed  or  a 
computed  proportionality  factor. 

The  simulated  programs  had  block  lengths  that  were  determined  by 
weighted  sums  of  normal  distributions,  and  block  lifes  that  were  expo¬ 
nentially  distributed.  An  efficiency  factor  was  computed,  where 

_  _ mean  number  of  active  words  _ 

mean  number  of  active  words  plus  inactive  words 

The  results  of  the  simulations  showed  that  efficiences  as  high  as 
0.80  can  be  obtained  for  some  types  of  programs  considered.  The  average 
efficiency  for  all  types  of  programs  simulated  was  Q.63.  More  important 
is  the  insignificant  difference  in  efficiencies  between  the  various  pro¬ 
cedures  for  a  given  program.  Hence  one  concludes  that  the  most  easily- 
implemented  procedure  is  the  one  to  be  preferred. 
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Another  question  of  interest  is  the  effect  of  page  size  on  the 
efficiency  of  these  procedures.  This  was  evaluated  by  simulating 
several  procedures  with  a  program  which  had  Poisson  distributed  block 
lengths  and  exponentially  distributed  block  lives.  The  mean  block 
length  and  life  were  selected  so  that  the  average  number  of  active 
words  was  relatively  constant. 

As  the  ratio  of  length  to  page  size  approaches  zero,  the  average 
efficiency  of  these  procedures  stabilizes  at  approximately  0.85*  The 
boundaries  created  by  pages  do  reduce  the  efficiency  when  the  page  size 
is  less  than  several  times  the  average  request  length.  However,  with 
pages  as  small  as  two  times  this  average  request  size,  the  efficiency 
is  only  reduced  by  approximately  twenty  percent. 

These  results,  and  extensions,  are  contained  in  a  forthcoming 
laboratory  report  by  J.  R.  Jump. 

3 . 3  Graph  Model  of  Concurrent  Computation 

It  is  necessary  to  formulate  languages  in  which  parallel  computa¬ 
tions  can  conveniently  be  described  and  to  have  methods  of  determining 
how  much  concurrency  the  intrinsic  structure  of  such  a  computation 
allows.  Karp  and  Reiter  [a  120  have  studied  and  generalized  a  model  for 
a  certain  class  of  parallel  computations.  This  model,  originated  by 
Karp  and  Miller  [b24],  represents  a  computational  algorithm  as  a  finite 
labelled  directed  graph.  Each  node  represents  an  operation,  and  each 
branch  represents  a  first-in,  first-out  queue  of  data.  The  sequencing 
of  the  calculation  depends  on  four  parameters  (each  nonnegative  integer) 
associated  with  each  branch.  For  a  typical  branch  p,  directed,  let  us 


say,  from  node  i  to  node  j,  the  parameters  are  as  follows; 


the  number 


J<8 


of  data  words  on  the  branch  at  the  beginning  of  the  calculation;  U^,  the 

number  of  words  placed  on  the  queue  whenever  the  operation  0^  associated 

with  node  i  terminates;  W^,  the  number  of  words  removed  from  the  queue 

whenever  the  operation  0.  initiates;  and  T  (T  >W  ).  a  threshold  giving 

j  P  P  -  P 

the  minimum  queue  length  that  must  be  reached  before  0^  initiates.  Each 
node  initiates  when  the  number  of  words  on  each  of  its  input  branches  is 
at  least  equal  to  the  corresponding  threshold.  With  each  node  is  asso¬ 
ciated  a  fixed  function  specifying  the  manner  in  which  the  inputs  deter¬ 
mine  the  outputs.  Because  of  the  first-in,  first-out  queue  discipline  and 
the  absence  of  conditional  transfers,  not  all  calculations  can  be  expressed 
as  computational  graphs.  It  appears,  however,  that  a  large  class  of 
iterative  calculations  can  be  so  expressed.  The  computational  graph 
model  is  asynchronous,  in  the  sense  that  the  speeds  of  the  operations 
associated  with  the  nodes  are  variable  and  unspecified.  Thus,  very 
many  sequences  of  operation  are  consistent  with  a  given  graph.  Never¬ 
theless,  it  is  shown  in  fb24  ]  that  the  results  computed  in  all  these 
sequences  are  the  same;  i.e.  computation  graphs  are  determinate.  Theorems 
are  also  given  in  [b24  ]  which  characterize  the  computation  graphs  that 
represent  terminating  computations,  and  provide  an  analysis  of  the  growth 
Of  data  queues. 

A  principal  goal  of  the  present  effort  is  to  extend  the  above  model 
as  far  as  possible  while  preserving  determinacy.  It  is  shown  that 
determinacy  is  preserved  in  certain  cases  even  when  the  operation  asso¬ 
ciated  with  a  node,  as  well  as  the  parameters  associated  with  the  branches, 
are  variable.  Also,  a  c -jnplete  analysis  of  termination  and  queue  growth 

is  carried  out  for  the  ;  articular  case  in  which  the  parameters  U  and  W 

p  p 


vary  periodically. 
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In  [al3  ],  a  synchronous  version  of  the  computation  graph  model  is 
studied.  An  execution  time  t^  is  associated  with  each  node  i,  and  a 
determination  is  made  of  the  fastest  periodic  rate  at  which  a  computa¬ 
tion  so  described  can  proceed.  This  rate  may  be  viewed  as  an  absolute 
limit  on  the  parallelism  possible  in  a  computation. 

This  generalized  model  is  better  suited  as  a  description  of  cer¬ 
tain  classes  of  computations,  particularly  iterative  ones.  As  an  exam¬ 
ple,  the  computation  graph  for  matrix  multiplication  is  considerably 
simplified  under  this  general  model. 

The  following  problems  have  been  considered,  (l)  Let  G  be  a 
strongly  connected  computation  graph  in  which,  for  all  branches  d  ,  we 

have  T  =  U  =  W  =  1.  We  wish  a  proper  execution  of  G  which  is  periodic 
y  ir 

in  the  sense  that  if  a  node  n.  first  initiates  at  time  t.,  it  will 

J  0 

initiate  thereafter  at  times  t  +  it,  t.  +  2n,  ...,  where  it,  the  period, 

0  J 

is  the  same  for  all  nodes  of  G.  Clearly,  if  the  computation  is  to  be 

controlled  by  a  clock  signal,  such  a  proper  execution  is  desirable. 

Such  an  assignment  of  times  t.  to  nodes  n.  is  given.  Moreover,  with  the 

d  d 

proper  choice  of  it,  the  computation  proceeds  at  a  maximal  asymptotic 
rate.  (2)  Suppose  the  frequency  of  the  clock  signal  controlling  the 
initiations  of  the  nodes  of  G  is  a  priori  specified.  Then  we  wish  the 
tj 's,  and  it  of  (l)  to  be  integers.  It  turns  out  that  if  it  is  not  an 
integer,  the  schedule  of  (l)  will  not  yield  integer  initiation  times. 

To  aid  in  the  investigation  of  a  periodic  schedule  in  this  case,  the 
so-called  free  running  execution  has  been  studied.  This  is  the  maximal 
rate  of  execution  of  a  synchronous  computation  graph  (one  whose  initia¬ 
tion  times  must  all  be  integers).  The  major  results  are  as  follows: 

(a)  Let  t?(t)  be  the  vector  of  branch  weights  of  G  at  time  t. 


50 


Then  there  exist  t',\  both  integers  such  *hat  for  all  t  >  t', 

^(t+\)  =  ^(t) . 

(b)  Let  Otj  be  the  number  of  initiations  of  node  n^  in  the  period 
it,  t+X],  t  >  t'.  Then  0^  is  independent  of  j,  Oj  =  a. 

<°>  a  =  51  =  max  .  _ 

loops  of  G 

where  £  t  is  the  sum  around  a  loop  of  G  of  the  execution  times 
of  the  loop  nodes  and  £  A  is  the  sum  of  the  initial  numbers  of 


data  items  on  the  loop  branches. 


Fran  these  results  we  infer  that  each  node  n  of  a  synchronous 

J 

computation  graph  under  a  free  running  execution  initiates  ultimately 
at  times 


tj,  t}.*,  +  ... 

2  2  2 

^ j  +  +  * '  ’ 


a  a  a 

V  ^  ••• 

re 

11  -  <  X]  + 

(3)  For  a  synchronous  computation  grapr.  we  sought  an  assignment  of 
integer  and  periodic  initiation  times  to  the  nodes  of  G.  In  the  case 
that 

/■£  1 1 

n  =  max  '•fir' 
loops  of  G 

is  not  an  integer,  this  is  impossible.  However,  the  following  assign¬ 
ment  is  possible.  Let  n  =  g.  We  can  find  integers  t^,  A^,  A^,  ...,  A^ 
such  that  node  n^  can  initiate  at  times 
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t.^  +  \j  +  2\f  •  •  • 

t^  +  A^,  t^  +  A^  +  \,  t^  +  A^y+  2\,  . .., 

ty  +  A?'1,  t.  +  A?*1  +  t.  +  A*?*1  +  2k.  . .., 
ii^ii  '  i  i  ’  ' 

This  assignment  is  such  that  G  computes  at  the  maximal  asymptotic  rate. 

It  would  be  of  interest  to  carry  out  further  studies  in  which  the 
computation  graph  model  is  generalized  to  accomodate  a  wider  class  of 
computations .  Consideration  should  also  be  given  to  the  problems  con¬ 
nected  with  programming,  for  a  given  multiprocessor,  a  comp-,  .ation 
specified  by  a  computation  graph.  In  particular,  the  following  ques¬ 
tions  arise: 

(1)  If  we  view  each  node  of  a  computation  graph  as  a  computer 
capable  of  performing  any  of  the  node  operations  of  G,  what 
is  the  minimum  number  of  such  computers  required  under  the 
various  schedules  of  the  problems  (l),  (2),  and  (3)  Just 
listed?  More  generally,  if  we  partition  the  nodes  of  G  as 
to  their  functions,  what  are  the  minimum  numbers  of  nodes 
of  each  type  required? 

(2)  The  allocation  of  storage  registers  for  data  queues  and 
their  utilization  so  as  to  minimize  the  number  required. 

(3)  Scheduling  problems  of  the  following  type:  Suppose  a  compu¬ 
tation  graph  G  requires  mi  nodes  of  type  i,  1*1,2, ...,n 

but  only  are  available.  How  can  be  constrain  G  so 

as  to  utilize  Just  nodes  of  type  i,  i=l,2, ...,n  in  an 
optimal  fashion;  i.e.  at  a  maximal  computation  rate  con¬ 
sistent  with  the  constraints? 

(U)  Programming  for  such  a  computer  system. 
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(5)  The  synthesis  of  a  computation  graph  given  a  computation  to 
be  performed. 

3.1*  Classification  of  Machines  and  Problems 

It  is  desirable  to  have  measures  of  problem  complexity  and  machine 
capability  which  permit  relative  comparisons  between  different  machine 
algorithm  configurations.  This  is  particularly  difficult  because  it 
is  necessary  to  determine  the  proper  weighting  to  be  given  to  the  variovs 
performance  parameters  such  as  cost,  rate  of  computation,  size,  con¬ 
venience,  etc.,  which  must  be  considered  in  the  choice  of  a  given  machine. 
The  real  world  situation  is  further  complicated  because  it  is  usually 
not  possible  to  specify  a  single  important  problem  for  a  specific  machine 
since  in  the  usual  situation  a  given  machine  is  assigned  many  types 
of  problems.  For  a  specific  algorithm  it  is  possible  to  determine  the 
maximum  degree  of  parallelism,  the  minimum  number  of  concurrent  time 
steps  required,  the  maximum  number  of  time  steps  required  and  the  maxi¬ 
mum  amount  of  storage  required.  A  different  algorithm  which  computes 
the  same  problem  may  have  different  requirements.  For  any  specific 
machine  problem  configuration  there  is  an  optimum  algorithm  which  is 
dependent  on  the  machine,  the  problem,  and  the  weights  assigned  to 
the  performance  function.  Comparison  of  machines  in  a  real  environment 
Is  difficult  because  the  performance  function  and  the  optimum  algorithm 
are  poorly  defined. 

We  avoid  the  specification  of  any  performance  weighting  function 
which  is  at  best  a  subjective  process  and  cannot  be  handled  in  an  abstract 
study.  However  the  parameters  of  the  performance  function  can  be  treated 
on  an  objective  basis.  This  requires  that  machines  be  specified  in  terms 
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of  gross  characteristics.  If  the  characterization  of  the  machine  is  too 
fine  then  it  is  impossible  to  obtain  a  tractable  abstraction  and  if  the 
characterization  is  too  gross  then  the  abstraction  is  trivial. 

It  seems  reasonable  to  first  determine  the  effects  of  concurrency 
and  parallelism  on  the  rate  of  computation.  If  a  single  sequence  machine 
executes  a  given  algorithm  in  T  time  units,  can  m  machines  execute  the 
algorithm  in  less  than  T/m  time  units?  Further  qualifications  are  neces-' 
sary  before  a  meaningful  answer  can  be  obtained  for  this  question.  Con¬ 
sider  the  usual  flow  diagram  language  used  to  describe  algorithms  for 
digital  computation.  The  flow  diagram  consists  of  substitution  state¬ 
ments  which  represent  the  direct  computation  and  alternation  statements 
which  effect  program  control.  A  step  in  the  computation  is  defined  to 
be  the  time  period  required  to  execute  a  statement.  We  can  now  ask 
whether  a  multi -machine  configuration  can  reduce  the  totel  number  of 
statements  of  steps  required  to  execute  a  given  algorithm.  A  program 
for  a  multi-machine  processor  consists  of  a  number  of  ordered  program 
segments.  The  segments  are  ordered  in  the  sense  that  a  given  segment 
requires  input  data  computed  by  other  segments.  However,  unless  there 
exists  a  storage  or  time  limitation,  each  of  these  segments  can  be  exe¬ 
cuted  in  some  sequence  by  a  single  machine.  Thus  a  multi-machine  con¬ 
figuration  does  not  reduce  the  number  of  algorithm  steps  required  to 
specify  a  given  algorithm.  A  particular  example  of  interest  is  look 
ahead.  A  second  machine  can  be  used  at  alteration  points  in  the  program 
to  achieve  look  ahead.  This  results  in  an  increase  in  the  number  of 
algorithm  steps  executed.  A  single  sequence  machine,  under  the  assump¬ 
tion  of  equal  probability  at  n  alternation  points,  will  on  the  average 
require  n/2  additional  look  ahead  computations.  The  multi-machine 


configuration  will  compute  n  look  ahead  steps,  one  for  each  alteration 
point . 


A  reduction  in  the  time  required  for  the  computation  is  obtained  by 
executing  time  independent  sequences  concurrently  using  a  multi-machine 
conf iguration  and  by  reducing  the  time  period  required  to  execute  a 
given  statement  by  the  employment  of  concurrent  or  overlapping  micro- 
operati ons  within  a  given  computer.  If  T^  and  T^  are  the  average  times 
required  for  non-look  ahead  and  look  ahead  computational  steps  respec¬ 
tively,  and  if  there  are  N  non-look  ahead  steps, then  the  multi-machine 
computation  period  Tm  in  terms  of  the  single  sequence  period  T  is 


T 


m 


nr. 


1  + 


Nr, 


1  + 


nT, 

C 

2Nr~ 


T  >  T 
m  m 


The  optimum  multi-machine  configuration  minimizes  the  time  required 
for  the  computation  and  the  number  of  machines  required  to  obtain  the 
computation  in  the  minimal  time  period.  It  is  clear  that  non-optimal 
assignments  exist  which  employ  more  than  the  required  minimal  number  of 
machines  such  that  the  number  of  machines  can  be  reduced  without  chang¬ 
ing  the  time  required  for  the  computation.  In  general,  it  is  not  possible 
to  achieve  100  percent  activity  for  all  machines  during  the  computation 
period.  The  actual  number  of  machines  which  can  be  used  is  a  function  of 
the  state  of  the  computation. 

The  control  of  the  multi-machine  configuration  adds  steps  to  compu¬ 
tation  algorithm.  Thus,  when  no  space  or  time  limitations  exist  it  is 
not  possible,  even  under  the  most  ideal  circumstances,  to  reduce  the 
computation  period  of  a  multi -computer  of  m  single -sequence  machines  to 
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less  than  1/m  of  the  period  required  by  a  single  sequence  machine. 

The  previous  discussion  suggests  the  folloving  problems:  Studies 
of  the  quantity  of  control  information  required  for  multi -computer 
operation.  Studies  of  finite  autonata  with  either  space  or  time  limi¬ 


tation  or  both. 
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4.  Combinatorics  and  Switching  Theory 
4.0  Introduction 

Most  of  the  effort  under  this  contract  is  being  expended  on  "macro¬ 
scopic"  systems  questions,  e.g.  problems  of  organization  and  control,  and 
of  the  utilization  of  storage.  Effort  is  also  being  given  to  questions 
of  switching  .circuit  synthesis  which  are  "microscopic"  in  character. 

The  justification  for  carrying  on  this  latter  type  of  research  is  two¬ 
fold.  First,  it  is  not  always  possible  to  totally  separate  the  macro¬ 
scopic  issues  from  the  microscopic  ones.  An  intelligent  evaluation  of 
the  practicality  of  a  new  computer  organization  usually  requires  an 
integrated  outlook,  which  takes  into  account  the  fine  structure  of  the 
system  as  well  as  its  larger  outlines. 

Secondly,  switching  circuit  theory  is  eclectic,  drawing  upon  such 
mathematical  topics  as  graph  theory,  matroid  theory,  semi-group  and  group 
theory,  linear  and  dynamic  programming,  and  classical  combinatorial  anal¬ 
ysis.  Techniques  which  are  useful  for  solving  switching  circuit  synthe¬ 
sis  problems  are  usually  sufficiently  general  to  be  useful  for  solving 
a  variety  of  other  combinatorial  problems,  including  many  having  to  do 
with  macroscopic  systems  questions.  Examples  include  combinatorial 
problems  arising  in  the  scheduling  of  multi-processor  systems,  the  static 
and  dynamic  allocation  of  storage,  and  the  assignment  of  capacities  to 
data  channels  for  maximum  performance/cost  ratio.  It  should  thus  be 
clear  that  our  work  in  this  area  emphasizes  basic  "combinatorics"  at 
least  as  much  as  "switching  theory." 

Our  program  in  this  area  falls  under  the  following  project  headings: 
(l)  Covering  Problems,  \2)  Threshold  Networks,  (3)  Cellular  Logic,  (4) 
Sequential  Circuit  Synthesis,  (5)  Miscellaneous  Problems.  Project  (l) 
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is  largely  completed,  and  it  is  expected  that  project  (2)  will  be  com¬ 
pleted  in  early  1 966.  Projects  (3)  and  (1)  are  still  in  the  formative 
stage.  A  description  of  each  of  these  projects  follows. 

4,1  Covering  Problems 

It  is  generally  acknowledged  that  the  central  combinatorial  problem 
of  switching  theory  is  the  so-called  covering  problem  [all],  which 
takes  the  form: 

Minimize 

n 

“  ■  Z  c3xj  (1) 

J=1 

subject  to 

Ax  >  1  (2) 

and 

x^.  =  0  or  1  ( j  =1, 2, . . . ,n) ,  (3) 

where  A  is  a  (0,l) -matrix.  (Each  element  a  of  A  is  etcher  0  or  l) . 

J 

This  problem  arises  in  exactly  this  form  in  the  second  phase  of  the 
Quine  method  of  Boolean  minimization,  and  variants  of  it  occur  in  a 
wide  variety  of  other  combinatorial  contexts. 

Research  on  covering  problems  has  included: 

(a)  The  identification  of  equivalent  problems  and  special 
cases,  and  of  relations  between  them. 

(B)  The  investigation  of  the  "graphic”  covering  problem, 
in  which  each  column  of  A  contains  no  more  than  two 
nonzero  entries. 

(C)  The  discovery  of  various  duality  relations,  which 
hold  for  covering  problems  generally,  and  of  a  new 
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method  of  solution  suggested  by  these  relations. 

Results  under  heading  (A)  will  be  contained  in  a  forthcoming  report 
tentatively  entitled,  "Coverings,  Packings,  and  Euler  Lines."  These 
include  the  following: 

(i)  consider  constraints  of  the  form 

Ax  >  b  (2!) 

and 


x.  =  nonnegative  integer 

where  A  and  b  contain  arbitrary  nonnegative  elements. 

Any  problem  of  this  apparently  more  general  form  is  act¬ 
ually  equivalent  to  a  covering  problem  with  constraints 


(3’) 


of  the  form  (2)  and  (3) • 

(ii)  For  every  "covering"  problem  of  the  form: 
Minimize  cx 

subject  to  Ax  >  b. 


x.  =  C  or  1, 

0 

there  is  a  complementary  "packing"  problem; 
Maximize  cx 


Ax  <  b 

=  0  or  1. 

The  feasible  and  optimal  solutions  of  the  two  problems  are 
in  one-one  correspondence  under  the  relation  x  =  1-X. 

(iii)  We  say  that  a  covering  problem  or  packing  problem  is  "graphic" 
if  its  matrix  A  contains  no  more  than  one  nonzero  element  in 
each  column,  and  each  nonzero  element  is  1.  There  is  an  ob¬ 
vious  connection  between  these  problems  and  problems  which 
require  that  the  vertices  of  a  linear  graph  be  covered  with 
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its  edges  (that  at  least  of  the  chosen  edges  be  incident 
to  vertex  i)  or  that  the  graph  be  packed  with  its  edges  (no 
more  than  b^  of  the  chosen  edges  be  incident  to  vertex  i) . 
Graphical  covering  problems  and  packing  problems  are,  of 
course,  complementary,  in  the  sense  of  (ii) .  They  are  also 
equivalent  to  problems  in  which  it  is  required  to  find  a  min¬ 
imum-length  tour  which  will  pass  through  each  edge  of  the  graph 
at  least  once.  This  latter  problem  is  solved  by  choosing  a 
set  of  edges,  such  that  when  duplicated,  an  Euler  line  exists. 
Edmonds  [bll]  calls  this  problem  the  Chinese  Postman's  Problem. 

A  special  case  of  the  Chinese  Postman’s  Problem  is  the  following. 

What  is  a  minimum- length  input  sequence,  such  that  a  given  sequential 
machine  will  be  forced  through  each  possible  state  transition  at  least 
once?  This  is  certainly  a  question  of  basic  importance  in  the  diagnosis 
of  malfunctions  in  sequential  circuits. 

Results  under  heading  (B) —investigation  of  graphic  covering  prob¬ 
lems — are  mainly  in  the  explication  and  simplification  of  certain  unpub¬ 
lished  solution  methods  due  to  Edmonds.  Edmonds  devised  an  algebraically 
bounded  computation  for  the  Chinese  Postman's  Problem  (a  computation  whose 
length  grows  only  algebraically  with  the  number  of  vertices  in  the  graph) 
based  on  a  computational  method  for  packing  problems  which  he  calls 
"matching"  problems.)  These  methods  are,  in  turn,  closely  related  to 
methods  for  solving  the  "shortest  route"  problem.  Problems  based  upon 
directed  graphs  can  also  be  handled  without  difficulty. 

Work  under  heading  (c) — duality  relations — has  been  carried  out  in 
conjunction  with  National  Science  Foundation  Grant  GP  2778,  "Partitioning 
Methods  for  Combinatorial  Optimization."  A  few  fundamental  identities 
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will  serve  to  illustrate  the  duality  properties  in  question.  Define 

Cov  A  (the  set  of  covers  of  A)  and  Cl  A  (the  closure  of  A)  as  follows: 

Cov  A  =  (x  )  Ax  >  1,  x .  =  0  or  l) 

J 

Cl  A  =  (a'  |  a'  >  A^  ,  for  some  i,  and  aj  =  0  or  l] . 

Then,  for  all  A,B,  such  that  A  =  Cl,  B  =  Cl  B, 

Cov  Cov  A  =  A 

Cov  (A  U  B)  =  Cov  A  fl  Cov  B 

Cov  (A  fl  B)  =  Cov  A  U  Cov  B. 

These  relations  may  be  compared  with  involution  and  DeMorgan's  laws  for 
sets: 

l  =  A 

(A  U  B)  =  A  H  § 

(aUTb)  =  A  U  B. 

Computational  methods  suggested  by  these  relations  have  been  pro¬ 
grammed  for  The  University  of  Michigan  IBM  7090  computer,  and  are  cur¬ 
rently  being  tested.  A  previously  issued  technical  report  [all]  contains 
all  theoretical  results  to  date.  This  report  has  been  accepted  for  pub¬ 
lication  in  the  SIAM  Journal. 

4.2  Threshold  Networks 

Work  on  networks  of  threshold  elements  is  being  carried  out  as  the 
Fh.D.  thesis  project  of  R.  Gonzalez,  and  should  be  completed  by  early 
1966.  This  work  is  significant  because  of  the  new  insights  it  contrib¬ 
utes  to  the  theory  of  linear  inequalities,  to  nonlinear  programming,  and 
to  "adaptive"  networks  of  threshold  elements,  of  the  general  type  often 
proposed-for  pattern  recognition  problems.  Research  has  included: 
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(i)  The  investigation  of  a  "dual"  method  of  elimination  for 
solving  systems  of  linear  inequalities. 

(ii)  The  study  of  minimal  synthesis  of  two-level  threshold  net¬ 
works,  by  an  approach  analogous  to  that  used  in  the  Quine 
method  for  minimal  AND-OR  synthesis. 

(iii)  The  synthesis  of  economical  "universal"  networks  which 

2n 

are  capable  of  realizing  any  one  of  the  2  switching  func¬ 
tions  of  n  variables  by  varying  weights  and  thresholds. 

(iv)  The  synthesis  of  universal  networks  for  restricted  families 
of  switching  functions,  e.g.  functions  for  which  there  are 
many  "don't  cares." 

Investigation  of  the  dual  elimination  method  has  shown  that  it  is 
actually  equivalent  to  the  method  of  "double  description"  due  to  Motzkin, 
Raiffa,  Thompson,  and  Thrall  (b29),  but  much  more  easily  derived.  The 
importance  of  the  method  in  the  present  context  is,  of  course,  that  it 
provides  an  efficient  method  for  testing  the  consistency  of  systems  of 
linear  inequalities,  and  thereby  resolving  the  issue  of  linear  separa¬ 
bility  for  a  given  switching  function. 

Dual  elimination  provides  the  backbone  of  the  synthesis  method 
mentioned  under  (ii),  and  reported  by  Gonzalez  and  Lawler  [a7  ].  This 
synthesis  method  is  a  two-phase  method,  just  like  the  Quine  method  for 
Boolean  minimization.  The  two  phases  are: 

(a)  generate  the  complete  set  of  "best  threshold  approximations" 
of  the  switching  function. 

(b)  select  a  minimal  subset  of  best  threshold  approximations, 
such  that  they,  together  with  one  additional  threshold 
element,  are  sufficient  to  realize  the  function. 
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Best  threshold  approximations,  of  course,  correspond  to  prime  implicanta, 
and  the  selection  problem  (b)  corresponds  to  the  ordinary  covering  prob¬ 
lem,  but  it  more  difficult  to  solve.  Algorithms  for  two-level  synthesis 
have  been  programmed  for  the  IBM  7090  computer,  and  are  currently  being 

t(J  GtC'd  i 

The  synthesis  of  universal  networks  relies  upon  the  application  of 
the  following  theorem  obtained  by  Qonzalez.  We  say  that  a  subset  of  ve». 
tiers  of  the  n-cube  is  total], y  linearly  soparab3,e  if,  for  every  possible 
partitioning  of  the  subset  into  two  parts,  there  exists  a  hyperplane 
which  effects  the  partition. 

Theorem 

A  given  subset  of  N,+  1  vertices  (x0,xi;xg, . .  is 
totally  linearly  separable  if  and  only  if  the  N  vectors 

*l”x0'  x2*x o'  xn'x0  "ro  lln“rly  independent  (the 

origin  x0  being  chosen  arbitrarily  from  among  the  subset 
of  vertices) . 

In  the  cane  of  problem  (iv),  the  synthesis  method  requires  that 
s  minimum-rank  partition  of  the  "cart"  vortloas  be  effected,  euch  that 
each  equivalence  oless  under  the  partition  ie  totally  linearly  separable. 
The  problem  thin  reduces  essentially  to  •  type  studied  in  matroid  theory 
[blO  ],  and  it  appears  to  be  possible  to  effect  e  direct  application  of 
known  results. 

t.'S  (X-lIular  Logic 

It  has  become  a  truism*  that  the  emphasis  of  switching  circuit  syn¬ 
thesis  should  be  changed  to  conform  with  the  demands  of  modern  integrated 
circuit  technology;  u.g.  component  counts  should  be  de-emphaulzed  and 


interconnections  should  be  given  primary  attention.  However,  it  appears 
that  there  are,  as  yet,  no  well-codified  design  requirements  of  the 
new  technology.  On  the  contrary,  the  situation  is  still  fluid  enough 
that  many  manufacturers  would  probably  be  willing  to  design  their  cir¬ 
cuit  layouts  and  interconnection  wiring  to  accomodate  a  reasonable  method 
of  logical  design. 

An  interesting  and  imaginative  approach  to  this  problem  area  is 
being  taken  by  Minnick  and  others  at  Stanford  Research  Institute  under 
the  name  "cellular  logic"  [b27] .  Under  this  system,  switching  functions 
are  realized  by  two-dimensional  arrays  of  cells,  where  each  cell  can 
realize  any  one  of  several  different  functions  of  two  variables,  simply 
by  cutting  the  appropriate  "outpoints"  in  the  cell.  All  connections  to 
these  arrays  are  made  at  regular  intervals  along  the  edges. 

A  related  approach  is  due  to  Canaday  [b5  ],  who  proposes  a  two- 
dimensional  array  of  3-input  "majority"  elements.  The  cells  of  Canaday' s 
arrays  are  simpler;  however,  he  requires  an  entirely  different — and 
probably  more  difficult— type  of  interconnection  wiring. 

Some  of  the  questions  which  these  proposals  suggest  are:  What 
growth  rates  are  necessary  for  the  dimensions  of  these  arrays  as  the 
number  of  switching  variables  increases?  Can  exponential  growth  be 
defeated  in  any  way?  What  trade-offs  are  possible  between  the  dimen¬ 
sions  of  the  arrays  and  interconnection  complexity?  Are  there  any 
advantages  to  be  gained  by  .. .,N -dimensional  arrays?  (Consider  the 
N-cube  arrangement  previously  proposed  for  the  Michigan  Iterative  Circuit 
Computer.)  How  should  sequential  circuits  be  realized  by  these  arrays? 
(Minnick 's  outpoint  logic  allows  each  cell  to  be  a  flip-flop,  but 


64 


systematic  design  methods  have  not  yet  been  proposed.)  What  techniques 
can  be  used  to  circumvent  the  delays  induced  by  many  levels  of  logic? 

4.4  Sequential  Circuit  Synthesis 

Recent  progress  in  the  decomposition  of  finite  automata  includes 
the  reformulation  and  simplification  of  the  Krohn-Rhodes  theory  by 
Zeiger  [b37],  and  new  results  on  the  reduction  of  feedback  loops  by 
Friedman  [bl3]  and  Brzozowski  [b3].  A  few  tentative  efforts  have  been 
made  to  apply  these  and  other  theoretical  results  to  the  synthesis  of 
sequential  circuits  from  a  restricted  set  of  simple  modules.  It  seems 
not  unreasonable  that  these  efforts  may  eventually  lead  to  efficient, 
systematic  design  methods  for  sequential  circuits. 

As  in  the  case  of  cellular  logic,  plans  for  this  project  area 
are  indefinite . 

4.5  Miscellaneous  Problems 

Some  minor  effort  has  been  devoted  to  combinatorial  problems 
other  than  those  described  above.  These  include: 

(l)  Optimal  encoding  for  the  discrete  noiseless  channel  with 
an  alphabet  whose  symbols  have  unequal  durations.  This 
work  was  originally  reported  on  at  the  International 
Conference  on  Microwaves,  Circuit  Theory,  and  Information 
Theory,  Tokyo,  Japan,  October,  1964  [&$],  and  has  row  been 
revised  and  new  tables  for  determining  bounds  on  optimum 
encodings  have  been  calculated. 
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(2)  Optimal  deferral  scheduling  for  multiple  channels  and 

linear  cost  functions.  This  is  an  extension  of  previous 
results  published  in  Management  Science  [  alO] .  It  is 
shown  that  a  dynamic  programming  method  of  solution  exists 
and  that  such  an  optimal  schedule  for  n  jobs  and  m  processors 
can  be  determined  by  an  amount  of  computation  which  grows 
as  nm.  This  result  depends  on  the  linearity  of  the  cost 


function. 
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